How to do science, but better
An interview with Adam Mastroianni
Welcome to Techno Sapiens! Subscribe to join thousands of other readers and get research-backed tips for living and parenting in the digital age.
Here at Techno Sapiens, we’re all about science that is accessible, useful, and even (dare I say it) fun. Few people do this better than Adam Mastroianni in his newsletter Experimental History.
To understand what Adam is doing with his Substack, and why it’s so different from traditional scientific writing, it’s useful to have some quick background on the way academic research typically happens. Here’s the standard model:
You do a study.
You write up the results in an academic paper. In order to get it published, you write this paper in a way that’s often 1) impenetrable to the average person and 2) kind of boring.
You submit the paper to a scientific journal.
The paper gets "peer-reviewed,” i.e., other scientists volunteer their time to read it and give you feedback.
You make changes to your paper based on that feedback
The paper gets published (often a year or more later).
The large publishing companies that own these journals require you to put your paper behind a paywall.
As a result, other people pay them to read about the study you did.
This results in lots of papers that are very difficult to read, if people can even access them at all.
Does this make sense? No, it does not. And that’s where Adam comes in. He’s using his newsletter both to translate research for the public, and to actually publish original research (instead of hiding it away in academic journals).
I hope you enjoy our conversation as much as I did!
Jackie:
Adam, I'm so excited we're having this conversation. Your newsletter is one of my favorites—every post somehow makes me laugh, changes the way I think, and teaches me something all at the same time. It also has the distinction of having the best subheadings (e.g. "THE CADAVER, THE BABY, THE SPAGHETTI SANDWICH" in this recent post and PEER REVIEW IS WORSE THAN NOTHING; OR, WHY IT AIN’T ENOUGH TO SNIFF THE BEEF in this one).
So, as a fellow academic psychologist who now spends (too much) time writing on the Internet, I’m excited to talk to you more about the new approach you’re taking to doing (and sharing) research through your newsletter.
But first, maybe we could just start with the basics—could you share who you are and what you do?
Adam:
I’m trained as a social psychologist and right now I teach negotiation at Columbia Business School, but here I’m a blogger. I take raw ideas and try to turn them into something useful by a) thinking about them for a long time, and/or b) gathering some data. They end up as essays like “Excuse me but why are you eating so many frogs” and research papers like “You’re probably wrong about how things have changed.”
Jackie:
It's definitely clear from your posts that these are ideas you've been thinking about for a long time—you have a really nuanced point of view that has changed the way I view certain things.
I'm curious, what do you think are some of the most interesting or surprising ideas you've written about?
Adam:
Here are just a few:
If intelligence = problem solving ability, you’d think that people with more intelligence would report having better lives. But there’s only a small and inconsistent correlation between IQ and happiness. ("Why aren’t smart people happier?”)
When people imagine how things could be different, they naturally imagine how things could be better. (“Things could be better”)
Only one of Einstein’s papers was ever peer reviewed. And overall, we’ve lost most of the diversity we used to have in science publishing. (“The rise and fall of peer review”)
Jackie:
Okay, I really want to come back to the peer review thing (!), but first, could you say a little more about the second one—this idea that people naturally tend to imagine how things could be better? What does this mean in practice? Why is that important?
Adam:
My friend Ethan and I ran a study where we asked people “How could [X] be different?” where [X] was all kinds of things: cars, “your life,” pets, YouTube, and so on. People typed out their answers, and then we asked them, “If it was different in that way, how much better or worse would it be?” On average, people told us that it would be better. So “How could cars be different? Well, they could run on solar power.”
This happened for every single item we tested. Even the items people said were already great! People loved their pets, but when we asked them how pets could be different, they said things like, “Well, they could not poop on the carpet.” Remarkably, 90% of people showed this bias toward better.
This effect didn’t depend on the wording of the question, and the same thing happened when we asked people about the stuff they had been thinking about recently. It also appeared in a sample of Polish people responding in English, and in a sample of Chinese people responding in Mandarin, so it’s not just something about being American or speaking English.
One reason this might matter is that this tendency may be the circuitry that runs the hedonic treadmill. We know that once people are pretty happy, it gets hard for them to get happier. Maybe that’s because no matter how good things get, people are naturally thinking of how they could be better.
You can read the whole paper here.
Jackie:
That is so interesting—basically, the reason that we're never satisfied with what we have is that we're automatically thinking about how it could be better. I wonder if that's related to perfectionism, too. Like, when we imagine how we, as people, could be different, we automatically think of ways we could be better—"well, I could spend more time with my kid" or "I could do a better job hiding my gray hairs" (totally hypothetically).
Okay, so when you link to that paper, you're actually linking to an article you wrote on Substack, which is kind of a different way of doing things in the whole "scientific papers" world, and this is one of things I'm most excited to talk to you about.
Your Substack offers a really interesting model of “public science” to bypass some of the issues with traditional scientific publishing. Could you say a bit about why you started your Substack and what your goals are for it?
Adam:
When I wrote scientific papers for journals, I felt like I was playing a game called “put together words in the right order so that my paper will get published.” It turns out that game is not only boring—it also feels terrible to do it, like I’m betraying my scientific principles. If I actually cared about discovering truth and sharing it with people, why would I write about it in dense prose and hide it in a paywalled journal?
Meanwhile, my friends Slime Mold Time Mold kept bugging me about starting a blog. When I tried it out, I suddenly felt so good. If you’ve ever seen Indiana Jones and the Raiders of the Lost Arc, there’s a scene where Harrison Ford has this staff with a ruby on top, and he sticks it into this 3D map on the ground, and the sun shines through at just the right angle and it illuminates the location of a treasure. I felt a little bit like that: in alignment with the universe.
At first, my Substack was just a place for essays, things that I was thinking about but would never be able to get published. Then, late last year, I figured: why not put one of my papers there too? I was writing the “Things Could Be Better” paper with my friend Ethan, and we couldn’t find any way of writing it for a journal that didn’t force us to lie in some way. We decided to tell the truth instead, write it all in plain language, and put it where anyone could access it. So now it’s on my blog.
Jackie:
I can definitely relate to the "put together words in the right order" feeling with scientific publishing!
I wonder if you could also lay out in more detail some of the problems with the current model of peer review you mentioned, so people can really get a sense for how what you're doing is so innovative.
Adam:
I think there are three important things to know about our current system of scientific publishing:
It’s very weird, historically. Until about 1960, there was a whole ecosystem of ways that scientists communicated their results. We’ve lost most of that diversity, and now virtually all of it happens the same way.
The system we use––universal pre-publication peer review––has extraordinary costs, but unclear benefits. It takes about 15,000 person-years of labor to review all the papers that get submitted every year, and yet reviewers don’t catch most of the mistakes in papers, or even blatant fraud. Not to mention that journals charge millions of dollar in subscription and submission fees every year (pay to get your article published, then pay to read it!).
Maybe we should try something else!
So that’s what I’m trying to do—revive some of the diversity we’ve lost. I’m publishing my work in the open, written so anyone can read it. Somebody’s gotta do it, so I guess it might as well be me.
Jackie:
This makes so much sense to me. I know one criticism you’ve gotten about this model—of doing and publishing science outside of academic journals and bypassing formal peer review—is that there’s no quality control. What if everyone starts using this model and random people start publishing poor quality (or downright false) “research” online? How do you typically respond to that?
Adam:
I totally understand this fear, and there’s a lot to say to it.
First, we would all like there to be an omniscient gatekeeper who makes sure that only true science gets published. Unfortunately, that gatekeeper doesn’t exist, and can’t exist. It sometimes takes years or even decades to figure out what’s true. It can take a considerable amount of effort just to make sure the data behind a paper isn’t totally fake.
Second, we don’t really have quality control right now; we just have the appearance of it. The most likely flaws in a paper are hiding in its data, but reviewers rarely check it, either because they don’t have time, or the necessary skills, or the data isn’t even available in the first place. When researchers deliberately insert errors into papers and send them to reviewers, reviewers miss most of them. Poor quality and downright false research gets published all the time. Whenever someone gets taken down for fraud, it’s almost always after that person has published tons of peer-reviewed papers, and I’ve never seen a case where it was a reviewer who caught them.
Third, the appearance of quality control without actual quality control is possibly worse than no quality control at all. When The Lancet, a prominent medical journal, published a tiny study suggesting a link between vaccines and autism, it gave huge firepower to the anti-vax movement—here’s a legit scientific source supporting our side! That paper was eventually retracted, but only a jaw-dropping 12 years later.
Fourth, we seemed to do all right without much gatekeeping for most of scientific history. If it was discovered before 1960, chances are it wasn’t peer reviewed. Einstein only ever had one paper peer reviewed, and he was so surprised and upset that he pulled the paper and published it elsewhere.
Fifth, and finally, if pre-publication peer review was critical to scientific progress, we should be able to see the difference it made as it became universal around the 1960s and 70s. Instead, indicators of scientific progress have ticked downward in the past 50 years, not upward. Interventions that have extraordinary costs should have extraordinary benefits; peer review costs a lot, but the benefits are so unclear that we can’t rule out the possibility that it actually makes us worse off.
Jackie:
This very much lines up with the bulk of my experience with traditional peer review. I’m convinced! [Also, 12 years!! Yikes!]
So then, a few more questions on the logistics—how/where will you do these studies? How will you fund them? What kinds of things are you planning to study?
Adam:
Often, constraints breed creativity. How can you run psychology studies if you don’t have a lab? One answer is you just run them online, which is where most psychology happens these days anyway. Or you go into the field. If you want to run a study on conversation, for example, why not partner with the singles mixer event happening at the local bar?
As for funding: fortunately, the stuff I’m interested in is pretty cheap to study. I think it’s possible to make enough money on Substack that I can fund the kind of research I want to do.
And for topics: one major advantage of this approach is that I don’t have to obey disciplinary boundaries. I have some research coming out soon on the Ideological Turing Test, which is where you have people who disagree try to pretend to be each other, and see how convincing they are. I’ve got some preliminary plans to build a more useful personality test. And I’d like to figure out whether flossing works or not.
Jackie:
Absolutely. Extremely interested in the Ideological Turing Test idea, but would argue the flossing study addresses a more urgent need.
Okay, one more important question before we wrap up:
On your Substack "About" page you note that prior to getting your PhD, you "got second place on a British reality show about cooking" and were "in a movie that currently has a 3% on Rotten Tomatoes." You've written about the latter (I could not stop chuckling to myself while reading it), but I am dying to know about the former. How did this happen? What was it like? What did you cook?
Adam:
I’ll write more about this someday, but I sort of ended up on reality TV by mistake. It was a show called Come Dine with Me and the premise was that four strangers take turns hosting dinner parties, everybody rates each other, and the person with the highest rating wins £1,000.
I thought it would be funny to apply for it, and then by the time I realized what I was doing it was too late. I cooked little pizzas, breakfast burritos (served “open face” because I couldn’t find big enough tortillas), and chocolate chip cookies. A guy on the show had a meltdown and kicked us all out of his house and then became a meme. The short version is: probably don’t go on TV, and definitely don’t yell at people on TV.
Jackie:
I will definitely keep that in mind. Thanks so much for doing this, Adam!
Adam:
Thanks for having me.
A quick survey
What did you think of this week’s Techno Sapiens? Your feedback helps me make this better. Thanks!
The Best | Great | Good | Meh | The Worst
In case you missed it
 | A guest post by
|
Hahahaha! I filled out the survey at the end. My response to the question "what can I do to improve technosapiens" was " I'm not looking for" better" 😂. When I clicked on submit the screen changed to "Thanks for making technosapiens better" 😂 😂 😂 😂 😂 😂 😂 😂 😂