Too Good to Be True: Bots and Bad Data From Mechanical Turk

Max Planck researcher writes about individual experience with Amazon MTurk

November 18, 2022

In her article in Perspectives on Psychological Science, doctoral researcher Margaret Webb summarizes her experience with Amazon Mechanical Turk (MTurk) and shares her results as a warning to others and call for caution.

Psychology today is moving increasingly toward digital sources of data, with Amazon’s MTurk at the forefront of this charge. In 2015, up to an estimated 45% of articles published in the top behavioral and social science journals included at least one study conducted on MTurk.

Webb recently conducted an online study via Amazon’s MTurk, eager and excited to collect her own data as a doctoral student for the first time. In her article Too Good to Be True: Bots and Bad Data From Mechanical Turk, she shares what she experienced using MTurk.

The researcher explains the eligibility criteria, the so-called consent quiz, and the completion process that participants had to go through to be part of the data collection. She also illustrates how she came across why the response time was unrealistic and in fact unusual. Following all this, she explains how she deduced that her sample was—at best—only 2.6% valid, according to her estimate. The results prompted her to write this as a warning: "it is indeed too good to be true”.

The article Too Good to Be True: Bots and Bad Data From Mechanical Turk was published in the prestigious bimonthly Journal Perspectives on Psychological Science.

Go to Editor View