Process tracing

Process tracing is a method used to evaluate and develop theories in psychology,^[1] political science,^[2] or usability studies.

In process tracing studies multiple data points are collected in comparison to simple input-output methods, where only one measurement per task is available.

Thinking aloud

Thinking aloud protocols are a type of verbal protocols, used for eliciting and analyzing verbal data. Once transcribed, the verbalizations can be categorized into a defined scheme. This makes the data manageable.

A thinking aloud protocol is one way to assess cognitive processes by letting people verbalize aloud what they currently think (concurrent verbal protocol) or what they were thinking (retrospective verbal protocol) while performing a task. Furthermore, depending on the type of the question asked, it can be distinguished structured verbal protocols and unstructured verbal protocols.

It is assumed that the report of this verbal stream can be used as a proxy for the content of working memory and hence reflects the cognitive processes while performing a task. Verbal protocols can be used in any research that focusses on understanding cognitive processes. Specifically, thinking aloud protocols can be helpful to study human decision making.^[1]

Historical background

The use of thinking aloud protocols was introduced into decision process analysis in the 1970s by Montgomery and Svenson. Since then, the method has continually developed and has made valuable contributions to decision research.^[1]

Procedure (Walker, 2004)

data collection

1) development of the task, based upon a hypothesis
2) instruction and training of the participant
3) recording audio of the performance

data reduction/content analysis

4) transcription of the verbalizations
5) definition of units of analysis (components, sequences, or complete models)
6) definition of exclusive and exhaustive encoding categories
7) encoding of the units of analysis into this scheme

analysis and interpretation

8) quantification of the prevalence of components
9) interpretation, in terms of the theory

The quality of verbal protocols in decision research

The quality of thinking aloud protocols depends on characteristics of the decision problem, as complexity, familiarity, importance, or whether singular or repeated. It is recommended to construct a decision problem with a medium level of complexity and importance and a low level of familiarity and little repetition.^[1]

Measurement criteria using thinking aloud protocols

One general problem with thinking aloud protocols is their validity (Russo, Johnson & Stephens, 1989). We cannot prove whether the participant really says what he is thinking. Particularly the validity of retrospective verbal protocols is debatable. Verbal protocols are vulnerable to non-veridicality. A verbal protocol lacks of veridicality if it fails to adequately represent the process that should be described. Errors of omission and errors of commission are typical reasons for a non-veridical measurement. Errors of omission happen if thoughts that are part of the process are not mentioned. Errors of commission happen if the verbal protocol contains statements about thoughts that were not actually part of the process.

Another important problem is the potential reactivity of concurrent verbal protocols . A verbal protocol is reactive if the verbalization process changes the cognitive process (Russo, Johnson & Stephens, 1989). This is particularly problematical when asking the participant why he’s doing something.

Finally thinking aloud protocols have the problem of reliability because of possibly biased interpretation of the researcher. Statements about the participants „thinking steps” have to be transcribed and coded into a categorization scheme. This requires training in order to overcome the possibly biased interpretation of these verbalizations (Trickett & Trafton, 2009).

How to deal with these problems It is important to obtain a cross-validation of findings from verbal data, e.g. with physiological or neural correlates.^[1] Furthermore it is recommended to add a silent control condition. Only then can we assess the influence that verbalization has on the primary process. Finally, to establish the interrater reliability, more than one researcher should code and interpret the data.

Advantages and disadvantages of verbal protocols (Walker, 2004)

disadvantages

- time-consuming
- validity
- reactivity

advantages

- rich data source
- naturalistic settings
- insight into cognitive processes

Eye tracking

As eye fixations are “the” way that humans gather information and every decision requires the acquisition of information it becomes obvious that eye tracking is a way of investigating decision processes with a lot of potential. Yet the method is still underutilized.^[1]

Advantages of eye tracking are that it is possible to trace a lot of information used during a decision task, it can sometimes be applied as a substitute for working memory and because eye movement are not so well controllable and difficult to censor, they can be recorded nonreactively.

However, there are also problems one has to deal with when working with the eye tracking method. Most of times an eye tracker is not part of a standard lab and it is still expensive. Another, more functional problem is the interpretation of eye fixations: The data reveals where participants are looking, but not what they actually are thinking. Yet, this is exactly what we want to find out more about.

In cognitive psychology studies, the mostly concerned movements are saccades. During these movements no information can be acquired, because the vision is suppressed. This makes it very interesting especially for judgment and decision making tasks.

Objects of regard

Eye fixation data could be used to test predictions: For example that magic tricks work because of misdirecting the eye gaze and failing to actually see what happens even if it happens right in front of ones eyes. One can also compare eye fixations between two groups: between experts and novices or a healthy and clinical population. By looking at the distribution of fixations the investigation of display properties becomes possible. For example varying the size of products’ displays drives fixations’ amount, which is predictive for sales. This design does not enable causal hypothesis testing, however with the suitable design, the distribution of fixations could answer causal questions as well. Studies show that fixation frequencies are an indicator of relative importance: the more fixations on an alternative the greater the importance of this alternative. It has been shown that the frequency of fixations on an attribute can also be transformed into a rank order of the attributes The fixation order tells us something about the evaluation of objects. Particularly the first and last fixations can point out important features of the decision making process. Russo (1994) defined sequences of fixations as different stages. The fixations before any refixations are defined as an initial phase of orientation or screening. The last fixations should reflect the elimination of alternatives as a last checking phase prior to the announcement of ones choice.

Time Measures

The total time fixation as well as the total time of a sequence of fixations can reveal important aspects of the decision process. The mean fixation duration is an indicator of the processing depth or effort, and thus, reveal an important aspect of the decision process too. There is evidence ( Pieters and Warlop, 1999) that fixation duration on a chosen brand is longer than on a nonchosen alternative. There are mostly pair comparison and fixation duration is indicating that there are different checking processes (Russo & Leclerec, 1994). Open question: Duration of saccade may improve the accuracy of the total duration. Sequences of fixation There are within-alternative and between attributes transitions. The definition of by-alternative processing is a fixation transition within an attribute between alternatives. A single transition within an attribute but between alternatives is defined as by-attribute processing. There is evidence that an increase in involvement (heightened by giving participants whichever brand they chose) not only led to longer fixation durations but also to more by-brand and fewer by-attribute transitions. ( Van Raaij, 1977) There are research that focus on how to distinguish between these two processes, such as Russo & Dosher (1983). They defined by-alternative processing as sequence that all three attributes had to be fixated without interruption. A by-attribute comparisons required contiguous by-attribute evaluations of all three attribute. There is evidence that by-attribute processing is the preferred process for decision making (Arieli et al., 2009), even if it is not possible to decide accurately e.g. between two gambles without by-alternative processing. Eye fixations as complementary data Different kinds of verbal protocol and eye fixation as a complementary data were compared (Gog et al., 2005). Such analysis reveals more information than only a single method. Fixations as a monitor of attention Some experiments investigate the peripheral process. In such studies, it is essential to ensure that an individual gaze at a fixation point and to identify if the eye has moved. With the gaze-contingent stimulus alteration (McConkie & Rayner, 1975) it can be imposed whererver an individual is currently looking. With this method the impact of emotional words, when presented parafoveally (Calvo & Castillo, 2009) and the left hemisphere advantage for peripherally presented words (Jordan et al., 2009). Critics: - Eye fixation show where people are looking, but not what they are thinking. A clear interpretation is hardly doable as a fixation e.g. can occur to an elimination or to another consideration of other alternatives, or it can be learning or eliminating. - There are only a few JDM theories that specify a decision process at the level of fixations on individual alternatives or even smaller units such as individual attributes. (e.g., a theory that propose a more or less continuously developing decision process or another theory that addresses a model of the differentiation of value over time).

History: Eye tracking for information of cognitive activity of decisions. [History of Eye tracking is already described in Wikipedia] Russo investigated 1978 in different methods of process tracing, such as eye tracking, verbal protocol or information display boards. They assume that different methods have different advantages and disadvantages. Further, they propose that a combined usage of different methods may reveal more information of process tracing background than a single method. They define eye fixation as a method with high quality but mention the difficulty of a valid interpretation and the difficulty to isolate relevant cognitive parts of behaviour. A big step forward in process tracing with the method eye tracking is provided from Russo & Leclerc (1994).

Why use eye tracking to measure decision processes? One potential of the eye tracking method is that new theories, predicting behavior at the level of eye fixation might elicit, thus entail the possibility to find out more about decision processes. At the same time it brings up the question how process based theories should look like in the future

In the eye of eye tracking data, how should process based theories look like? Why move from simple theories to the complex detail of process-based theories? – depends on paradigm What kind of jdm processes might eye fixation data illuminate? Example: Identifying heuristics (1. Approximate one attribute to zero, either because its difference was small or its importance was negligible. 2. Count majority of confirming attributes) Data might prompt the devising of experimental designs and visual displays that could provide a new class of tests Investigation of already existing heuristics. Preference drives perception: People look more at what they prefer. – Will inducing people to look at a stimulus longer or more often increases their preference for it? Will attention drive preference?

Eye fixations are the primary way that humans acquire information. To be able to use this immense amount of data, JDM theories need to be developed that require such detailed information acquisition data.

Mouselab

Mouselab is a computer program that records information acquisition. The basic idea behind process techniques is the record of eye movements. Mouselab, as a computer-based technique, models eye movements by mouse movements. The information is presented in information cells which are ranged in a matrix. They are covered by an overlay, but can be opened by moving the mouse pointer across the information cell. During the time the cursor is in the cell, information can be acquired. Mouselab is harken back to the information board idea (e.g. Payne 1976). Here, the information is hidden in envelopes attached to a cardboard. To obtain information, a card has to be pulled out.

Designing

For setting up a Mouselab study, there are two programs which can be used online and offline (e.g. MouselabWeb; Willemsen & Johnson, 2008). The way information is displayed or labeled can have a huge influence on the decision. Presenting the information in a certain order can lead to effects which might be a result of display layout (e.g. people tend todward following the natural reading order, which influences the starting box). Counterbalancing the position of different types of information can help to antagonize ordering effects. Exact labeling of the cells can prevent exploratory search - interpretation of information acquisition is easier when the participants focused on the text in the cells. Information in the cells should not be too simple, because participants might memorize the content and therefore stop acquisition.

Process Data

The output of a MouselabStudy consists of hundreds of information pieces. Variables that are measured are: Time per trial, average looking at each box, the number of acquisitions, time between acquisitions. Unmotivated participants can be identified by unrealistically short trial times and unrealistically long ones or long times between acquisitions. About 5-10% of the observations have to be removed from analysis. Very short acquisitions, due to mouse movements, are filtered out (less than 200ms).

The process data can be presented by icon graphs (Johnson et al. 2002). The height of each icon is proportional to the number of acquisitions for the information cell. Icon size represents a measure of attention. With the help of arrows, transitions between boxes can be displayed.

Additional predictions can be made for reading and choice phases: The size of the boxes represents the amount of attention, participants give to the information cell. Arrows indicate direction and comparisons which are possible during information acquisition.

Process Tracing in Political Science

The method works by extracting all of the observable implications of a theory, rather than merely the observable implications regarding the dependent variable. Once these observable implications are extracted (particularly with reference to the microfoundations of how a theory's independent variable causes the predicated change in the dependent variable) they are then tested empirically, often through the method of elite interviews but also often through other rigorous forms of data analysis.^[3]

It is often used to complement comparative case study methods. By tracing the causal process from the independent variable of interest to the dependent variable, it may be possible to rule out potentially intervening variables in imperfectly matched cases. This can create a stronger basis for attributing causal significance to the remaining independent variables.^[4]

References

1 2 3 4 5 6 Schulte-Mecklenbeck, M.; Kühberger, A.; Ranyard, R., eds. (2011). A Handbook of Process Tracing Methods for Decision Research: A Critical Review and User’s Guide. New York: Taylor & Francis.
↑ Ford, J. Kevin; Schmitt, Neal; Schechtman, Susan L.; Hults, Brian M.; Doherty, Mary L. (1989). "Process Tracing Methods: Contributions, Problems, and Neglected Research Questions". Organizational Behavior and Human Decision Processes. 43 (1): 75–117. doi:10.1016/0749-5978(89)90059-9.
↑ King, Gary; Keohane, Robert O.; Verba, Sidney. Designing Social Inquiry.
↑ George, Alexander L.; Bennett, Andrew (2005). Case studies and theory development in the social sciences. London: MIT Press. pp. 214–15. ISBN 0-262-57222-2.