This may be a clearer answer:
Tobias Gerstenberg, assistant professor of psychology in the Stanford School of Humanities and Sciences, has been fascinated by questions of causality since graduate school. In addition to being a deep philosophical concept, causality plays an important role in many fields of study such as law, computer science, economics, and epidemiology. “I love causality because it matters for everything,” he says.
And causality could play a particularly important role in building a more humanlike AI, Gerstenberg says.
In recent research supported by Stanford HAI, Gerstenberg has shown that humans’ capacity for counterfactual simulation – thinking through what would have happened if a causal agent weren’t present – is critical for judging causation and assigning responsibility. “If AI systems are going to be more humanlike, they will need that capability as well,” he says.
Gerstenberg and his colleagues have taken a step in that direction by creating a computational simulation model that captures important aspects of how people judge causation and assign responsibility. In several contexts, the model can predict the likelihood that people will
assign causal responsibility to an object (such as a billiard ball) or
a social agent (such as a character in a computer game).
The journal
Trends in Cognitive Sciences recently invited Gerstenberg to write a review of his work on causal cognition.
That publication inspired the following conversation in which Gerstenberg describes what people should understand about causal cognition; how his research relates to AI; the risks and benefits of using AI to simulate causal scenarios for assigning blame in a legal context; and how that capability might lead to various ethical or societal problems.
What do you want people to understand about the nature of causal cognition?
I’m postulating that when people make causal judgments or assign responsibility, they’re not just contemplating what happened or what they saw. In fact, they are regularly going beyond the here and now to imagine how things could have happened differently. The process of thinking about counterfactual possibilities is key for explaining how people make causal judgments in both physical and social contexts.
In fact, my research shows that the same process of using counterfactual reasoning is going on if we’re judging
whether a billiard ball causes another billiard ball to go into a hole,
whether a block supports another in a stack, or
whether a person helps or hinders another person in a social setting.
You can think of counterfactual thinking as a domain-general ability that we have, and how it plays out depends very much on the domain that we’re applying it to. So, if I’m thinking about the physical domain – such as with billiard balls – my understanding of what would have happened in the absence of the balls colliding is driven by my mental model of how forces interact in the physical world. Whereas when I’m thinking about whether a person was helped by another person, I’m using my intuitive understanding of psychology and my mental model of how somebody would have acted in the absence of someone else’s help.
Another important point I’m making is that in prior work about determining whether a person is helping or hindering another person, researchers have been more interested in inferring people’s intentions than inferring causality. And there’s a difference between inferring whether somebody had the intention to help or hinder and judging whether they actually were helpful. I like to use the example of going grocery shopping with a small child who puts things into the basket with the clear intention of helping, but they’re not actually helping in the sense of making the shopping experience more efficient. In fact, it might have been easier without them. For judging the child’s intentions, I just look at their actions. But for determining whether they were actually helpful, I need counterfactuals.
So, while causality is a phenomenon that people have been interested in, they may not have recognized that part of what’s underlying this phenomenon is this process of counterfactual thinking. And I’m saying that a whole host of varied phenomena rely on the idea that we as humans build mental models of the world that allow us to imagine changing what happened, and then play out how such a change might have created a different situation. And these capacities are quite important for making causal judgments.
What does your research about causality tell us about AI?
If we want to develop AIs that in important ways emulate the way humans think about the world, they will likely need to be able to have these kinds of causal reasoning capacities too – mental models of the world that allow them to evaluate, step by step, how things might have played out differently.
For example, if we ask an AI, “Why did this happen?” and get an answer, we humans will interpret the explanation by imagining the counterfactual possibilities. Essentially, if the AI tells me that A caused B, we will understand that to mean that if A had not been the case, B would not have happened. But if the AI doesn’t share that same mental model – if it basically doesn’t use the phrase “why did this happen” in the way we humans do, and if it doesn’t have the ability to do counterfactual reasoning – it will not explain things to us in a way that will make sense to us.
AIs will need to understand causality at the right level of abstraction as well. For example, there’s a problem we call causal selection: In principle, a counterfactual account could go back to the Big Bang. And while the Big Bang is
a cause of everything, it’s also
the cause of almost nothing. In any given situation, there are a lot of things that could count as a cause. But the things that humans identify as
the cause or causes are a much smaller, more pragmatic subset.
In legal settings there are additional criteria for causation such as the
sine qua non or “but for” test, which is just a counterfactual: But for the defendant’s action of speeding through a red light, the plaintiff would not have been injured in the crosswalk. There’s already interest in using AIs to simulate alternative scenarios in these kinds of cases. And I’m trying to make the legal theories of causation a bit more precise by building computational models that predict how people judge causation – and helping with the translation of some of these ideas into AI.
If AIs do become adept at judging causation, how might that ability be either useful or problematic in a legal context?
Imagine a case where a car accident happened and the prosecution claims that it was caused by the defendant speeding. And suppose the prosecutors use AI to simulate a counterfactual scenario showing what would have happened if the defendant wasn’t speeding. The jury might find the simulation convincing (because seeing is believing), even though the prosecutors or defense attorneys could have constructed 100 different potential scenarios with different endings. What’s missing here is information about the uncertainty involved in generating the simulation. And that could be dangerous or misleading in a legal context.
On the other hand, we’ve suggested in a
recent paper that perhaps this problem could be overcome if each legal team had a legal fact finder who generates multiple counterfactual simulations and includes information about uncertainty. So, for example, the legal fact finder would testify that based on their simulations and video analysis, there’s a 98% chance that the car speed was a contributory cause.
Insurance companies might actually find this type of tool quite useful as well because they often deal with the question of fault. People are already doing 3D reconstruction using video from smartphones and CCTV cameras. It’s conceivable that in the not too distant future, claims adjusters could combine such 3D reconstructions with the simulation of counterfactuals to determine who’s at fault.
www.whitehouse.gov
kpmg.com
—> 
