Field experiments were once rare in economics but now are much more common, thanks in part to the work of University of Chicago professor John List.
The use of real-world observations in the dismal science instead of laboratory-style studies has been the focus of List’s work for years. He has also turned his attention to related questions around the reliability of experimental results, which are often exciting and increasingly influential on policy, but may turn out to be false positives or fail to scale up.
Speaking at the Behavioural Exchange conference in June, List discussed his efforts to develop a mathematical stress-test for experimental findings — essentially a formula that takes into account the number of independent replications of the results and allows their reliability to be expressed in terms of the probability of actually being true.
This he described as “the science of using science” and argued that a particular set of research evidence should be have 95% probability before it is accepted as the basis for new public policy.
For its latest podcast, the Behavioural Economics Team of the Australian Government (BETA) has just published a short interview with the professor it recorded while hosting the international conference.
“My experiments have essentially been around the questions like, why do people give to charitable causes?
“Why do women earn less money than men in labour markets? Why do inner city schools continue to fail? These broad based social questions have been the ones that I’ve tended to go after using field experiments.”
List talks about how he always enjoyed testing the limits of theory and proposing adjustments based on experiments out in the real world instead of in a lab setting, and now a lot more academics have since caught up.
“Back then, I was on an island, but at least I was there early on to see the trials and travails. A lot of people were telling me that I was dumb, that I should be doing lab experiments or I should be using naturally occurring data, but it always made sense to me to combine randomisation with realism, which is what field experiments do.
“It just felt right, that this is an important avenue to take and I think it will be an important avenue for decades to come.”
List also talks about his role as chief economist for Uber’s main competitor, Lyft, and his work around applying behavioural science to design performance-pay schemes for teachers that actually work as intended.
“We went to teachers in September, which is the start of the school year in America. We told them that, ‘Here is $4,000. We’re looking for you to value add for your students this year. We are going to test them again in June. If you value add at a very high level, you can keep your $4,000 and we might give you more.’
“But we told them, ‘If your students do not achieve on that test in June, you might have to give money back.’ It’s exactly like a typical bonus frame, except we move the payment to the start of the program rather than the end of the program.
“So then it’s sort of a baseline. We have another group of teachers that have the traditional scheme, which we tell the rules are exactly the same. We go to them in September and we say, ‘You are part of an experiment and you can make money. We’re gonna look at your students’ standardised test scores in next June, and then we’re gonna pay you cash.’
“We basically compare students in the clawback or the loss-averse treatment, along with the traditional bonus-scheme treatment. What we find is that the traditional bonus-scheme treatment really doesn’t work that well. It works about like a control group: what I mean by that is teachers who have no incentive pay.
“So all the critics are pretty much right. If you do a bonus scheme in the traditional way, it really doesn’t work. But if you use the clawback, or leverage loss aversion, it works splendidly.”
READ MORE ON THIS TOPIC:
▪ When nudge comes to shove: the e-health optout
▪ Cass Sunstein’s Bill of Rights for Nudging
▪ Why Martin Parkinson wants a second generation of behavioural economics
▪ Reflections from the founder of Australia’s national nudge unit
▪ New journal delves into behavioural approaches for public administration
▪ The OECD’s six principles for spreading behavioural insights across government
Any feedback or news tips? Here’s where to contact the relevant team.