In machine learning, we’re used to picking up where someone else left off. Download a model and a dataset, run the public code, change something, and try an idea.
But when I read an interesting social science experiment and want to change the wording or a condition, I often have to reconstruct the materials, conditions, procedure, and analysis first. Running it with people again also involves ethics review and recruitment. A small change becomes a whole new project.
My advisor recommended David Donoho’s Data Science at the Singularity, which discusses frictionless reproducibility. My takeaway: a field can move quickly when one person’s work becomes the next person’s starting point. I want that for human studies too.
I want the experiment to remain available after the paper is published: something others can open, inspect, and run again. With LLMs, we can also have agents play the participants, read the same questions, make the same choices, and try new conditions. The simulated results still need to be checked in human studies.
That’s why I started building HumanStudy-Hub: working with a community to turn published human studies into reusable, runnable experiments.
These experiments can also be benchmarks for agents, or a gym: keep the experiment fixed, change the model, prompts, or participant settings, then compare with human results. See where the behavior matches, where it doesn’t, and keep adjusting. HumanStudy-Bench is a step in that direction.