Saturday, May 29, 2021

Smell you later

A few weeks ago, I started a new job as a data scientist at the logistics division of a pharmaceutical company. This was primarily motivated by family reasons. My wife works at an art museum in Chicago and could not find satisfactory work in my small college town. We'd been living apart while I struggled to find a new professorship that might put us in a city she'd like. The academic job market is lousy enough that I was lucky to have a job anywhere -- I was not the kind of research superstar and granting rainmaker who could work wherever I pleased.

The school going online in 2020 made it possible for me to spend a fourth year in academia, but with a big naughty dog and our first child on the way, there was no way I could return to in-person classes in the fall. I flailed around on LinkedIn looking for a chance to get into data science, but later lucked out when a headhunter found my resume on Dice and scooped me up.

So now I'm working in industry. I am happy with this development: the pay is good, I don't have to drive two and a half hours to see my wife, and I get to turn my brain off at 5pm. 

My impact on the field

I'd like to think that I've managed to do a few useful things in my time in social psychology.

  • I've tried to point out that, contrary to some previous claims, there is reason to doubt that 15 minutes of violent video games causes a detectable increase in aggressive behavior. I've identified publication bias, and I've published null results.
  • I've tried to bring greater awareness to publication bias and meta-analytic ways to adjust for it. The paper I wrote with Evan Carter, Felix Schonbrodt, and Will Gervais has been pretty successful in this light and quickly became my most cited paper.
  • I've managed to get some really suspicious papers retracted. I'd like to think that this has helped shift the norms in post-publication review and retraction, if only slightly. I'm happy I could contribute a few more data points to the Retraction Watch database.
  • Paul Bloom once saw me order a Sazerac at SPSP, asked me what it was, then ordered one himself. So you might say I've influenced some of the big names.

The stuff we do doesn't matter

The thing that I'll probably most miss about academia is getting to research whatever I'm curious about. I'm not missing it just yet, however. While on the tenure track, I didn't think I was discovering hidden truths about the human mind by studying the American college undergraduate and Prolific.co participant. 

Honestly, I'd felt pretty discouraged about research for a while. The things we study tend to have small effects, and when we can't detect those small effects a second time, it can be hard to tell why. (Possible explanations include noise, publication bias, errors in methods, differences in populations due to culture or even the passage of time.) It's why we spend so much time arguing about the fidelity of replication methods and hidden moderators.

Because the things we study have small, purportedly delicate effects, it's rare that we expect to see them applied and working in the real world. It's unpleasant to say it, but I feel that a lot of the research that we do doesn't matter. It's because it doesn't matter that we were able to get all the way into the 2010s before having a replication crisis. If we had screwed up our basic science in physics or biology or chemistry, we would notice pretty quickly when the engineers told us their bridges were collapsing or the crops were dying or the soda pop was going flat. By comparison, very little in social psychology seems to be applied or expected to work in any routinely detectable way.

The lackadaisical response I've sometimes received when raising concerns about papers has further convinced me that most social psych research does not matter. When I email a journal to say "none of these statistics add up" or "these effect sizes are ridiculously big," I often get no reply. Compare this to the sort of all-hands-on-deck response we might get if we found poison in the dog food. It doesn't matter that the product is no good -- we produce it for the sake of producing it, quality irrelevant.

In comparison, the stuff I'm doing as a data scientist isn't glamorous, but it's useful. Some of our projects save the company millions of dollars a year in shipping costs. That's a lot of gasoline and traffic and cardboard and dry ice that we're able to save. Reducing the amount of oil and packaging that gets used up might be the most useful thing I've done in years.

I sometimes wonder if the future of social psychology isn't in industry. Academics are spending their little budgets on MTurk studies and undergraduate surveys while tech companies have terabytes of data on people's activities (real people! from the real world!) and can run A/B tests whenever they like. People sometimes complain that "the best minds of our generation are working to get people to click on ads," but it's also the case that the best datasets of our generation are also dedicated to the same cause.

I also wasn't confident that I was doing anything useful as a teacher. At some level it broke my heart to know that students were paying money to attend my classes and take my exams. I probably expected too much, but I felt like my classes somehow had to lift my undergrads into a job. I could see a computer science course doing that, but not social psychology. I often felt like a failure for not being able to fix what generations of poverty and decades of underresourced primary schools had done to my students. Yes, there were a few exceptional students that I was able to help get into a job or a graduate program, but I feel like if it weren't me, it would have been another professor.

The Thumos Treadmill

Liam Kofi Bright has a really interesting philosophy article out on why academics are motivated to do fraud. In it, he examines acadmics through Plato's tripartite model of the soul: epithumia, the materialistic element, associated with the masses; thumos, the honor and esteem element, associated with the armed forces; and nous, the reason and wisdom element, associated with philosophers and scientists. Plato assumes that scientists are motivated by nous; Bright argues that they are also motivated by thumos, which has both good and bad consequences.

The neverending quest for thumos kind of drove me nuts. I didn't have a strategy for finding a new job that didn't revolve around just publishing as much as I could and hoping it would be enough. (I probably should have written a grant at some point, but I was too busy writing papers and too discouraged by terribly low funding rates.) It felt awful to stretch myself trying to build a better CV, knowing all the time that no matter how good my CV was, I still might not get what I wanted.

I liked writing papers, but they always felt like ten times more work than they should have been. And once finished, the interesting ones made for arguments and headaches and the quiet ones sank into the literature without a sound. I will admit that sometimes arguments tempted with the promise of a possible comment or reply, yielding another line on the CV at lower cost than a three-study empirical paper. For all my high-mindedness, I was still subject to the same pressures as everybody else.

I'm done with thumos for now. I'm enjoying the comparatively relaxed pace of my new job. I no longer have to try to become a one-in-a-thousand genius so I can get hired at an urban university; I can just be a guy who does his job well enough to not get fired. Maybe in a few years I'll get hungry for promotion to senior data scientist or something like that, but it'll be a while, if it ever happens. I've always been willing to earn a little less money in exchange for working a little less.

Normalize leaving academia

Academia is nice if it works out and you like the work and flexibility enough to take the pay dip. However, academia isn't going to go out of its way to take care of you. It can barely take care of itself.

If you're good at coding and data analysis, you can probably increase your salary by 50-100% by going to industry. Money isn't everything, but it's not nothing, either. I hate to be so capitalistic, but money feels a little bit like respect. It puts a clear and concrete value on your skills in a way that the occasional citation does not. After so much time begging for a tenure track position, getting a single offer by incredible good fortune, and going through it all over again trying to move to the city, it's nice to feel wanted again.

Being prepared to leave academia has benefits beyond the materialistic. You might recognize some of my more pugilistic works pointing out effects that are artifacts of selective removal of outliers, or effects that are too consistent or too big to be true, or the last two years of the Zhang affair. It can hurt your academic career to make enemies or to be known as a trouble-maker. Even outside of that, these projects had an opportunity cost; while I was writing criticism, I was not doing my own primary research and making discoveries with my name on them. Being ready to leave gave me the freedom and power to criticize what I felt needed to be criticized.

Spending any amount of time on the Zhang affair would have been a career mistake, of course, had I planned to stay. People are grateful to you for cleaning up the mess, but getting some papers retracted isn't going to get an entire department to want to hire you and work with you for the rest of your lives. Error detection isn't yet sustainable as a primary research interest; if it was, Elisabeth Bik would be an endowed chair.

If the NSF ever wants to assemble people for a real data police with a real budget, let me know. Until then, I'm going to be over here, writing code, cashing checks, and raising my family.


I'd like to thank:
  • Bruce Bartholow, my PhD advisor, for trusting me enough to do my thing as a graduate student, even when it involved making trouble.
  • Laura King, who ran some good classes and an excellent journal club, and who was one of my letter-writers when I was looking for a job.
  • Jeff Rouder, who was crucial to my development as a scientist and Bayesian, and who introduced me to R, which is now how I support my family.
  • Daniel Lakens, whose early blog posts provided code for meta-analysis and PET-PEESE meta-regression, showing me that meta-analysis was just a single line of code, and not an arcane ritual requiring several degrees in rocket science. This opened a whole primary research interest to me.
  • Illinois State University, for running a good psychology department where people generally get along. I'd like to particularly thank my department chair, Scott Jordan, for running a department where expectations are both reasonable and clear and professors aren't encouraged to eat each other alive.
  • The Society for the Improvement of Psychological Science, particularly its early founders, for putting open science and post-publication peer review on the agenda. It's the only reason I was able to spend four years on the tenure track, and the only reason the science was worth doing.

In the roguelike community, people would post their "morguefiles" at the end of their game to show the thrilling way their character won or the tragicomic way they died. Here's mine:

        Joe_Hilgard the Assistant Professor (level 22)
             Began as a Social Psychologist
             Was a friend of SIPS.
             Escaped with a job
             ... and 31 publications on May 10, 2021!
             The professorship lasted 4 years.

Joe_Hilgard the Social Psychologist (HuSP)         

+3,+1 RStudio IDE {Int+4, Enhance}
+4 robe of Git {Dex+4 Int+2, Version Control}
+2 mug {Coffee}
+2 visored helmet "FunnelPlot" {Dam+3}
+2 gauntlets of emailing
+2 boots of BayesFactor
+1 ring of PubPeer
+2 ring of Monte Carlo

@: tired, grumpy, collecting data on college undergrads and MTurk workers, resistant to enchantments
}: 5 runes: meta-analytic, experimental, silver, iron, abyssal

You escaped.
You attended 4 universities and accumulated 22 years of education.
You visited the Abyss 2 times.
You visited 1 Labyrinth.

Your h-index was 18.
You got other authors to retract 5 articles.

You had 261 unread emails.
You owed your collaborators 5 overdue action items.
You signed your peer reviews.

        #.#######.#     #.#######
        #.........#     #........
        #......#.8#.##.#### #.#
        #......#<.........# #.#
                    #####.#     #
#####################.....#     #

Message History

You preprocess your data.
Ping! New email.
You hold Zoom meetings with students.
Ping! New email.
The fraudster publishes!
The fraudster publishes!
The fraudster publishes!
Publish which blog post? (* to show all)
People are talking about the blog post.
Ping! New email.
A fraudulent paper is retracted!
You hold Zoom meetings with students.
You hold Zoom meetings with students.
There is a job offer at a company here.
Are you sure you want to win?
You have escaped!


  1. I'm so happy I came across this today. I also recently signed a very nice offer outside of academia but then started getting cold feet and feeling like 'what if I'm making a mistake?' Reading this was very reassuring! Thank you.

  2. Joe, I think I am a bit jealous. I used to marvel how useless I could be and get paid for it. Now, even that is old and tired. Best, Jeff

  3. Joe, it was a pleasure to have met you at SIPS! As I really like you and your work, I feel a bit at a loss now. I wish you a lot of success in your private life and fun at work.

  4. I'm a psych student pivoting to data analytics in my undergrad. Thank you for your honesty. This post really gives me a lot to think about.

    I wish there were more incentives for academics who want to improve the statistical rigor of psychology....

  5. Joe, interesting blog! I'm 69 y/o and made a similar decision about 40 years ago. The good news - I could support a family on the money, and the kids genuinely appreciated it - no small reward. The only bad news is that I really didn't get to create interesting projects and break new ground unless it was directly related to a company goal, and obviously helpful to them in the near term. I think (but not sure) I made the wrong choice for me. I've always loved the scientific process. I've always craved to make fundamental advances in knowledge, even if they didn't deliver tangible value in the short run. One mistaken assumption I made in academia is that if your chosen sub-field or hypothesis doesn't prove to be correct, you will have wasted years of your life. I now believe that a good scientist will "pivot" quickly when they find a dead-end or a too low effect size. In your case, maybe a pivot from social science to biological science would have been more satisfying. I also now believe that I would strive to focus on positive findings (still hard for me). Failure in science is part of the process, and it is hard to maintain morale up for many of us (including me). Still, for me, "science" is a journey that inspires me after all these years.

  6. Hi Joe! Someone brought this blog post to my attention on twitter recently, just wanted to say this is very well written and insightful post about the issues in academia and the struggles of leaving it. Every point in this really resonated with me and my recent decision to leave. Thank you for speaking up and making it a bit more normal to leave!

  7. Just left academia, am also a social psychologist, made uncomfortable by what I saw in the academic system, AFTER being exposed to industry and NGO work. Good luck to us who chose an alternative career, post PhD.

  8. I have the same feeling that most social psych research is useless, but packaged up with a pretty bow to seem that it will have impact. Great to hear that I'm not alone in this view that is often very difficult to voice! I also find it very hard to get my head around the amount of money that is thrown at some of these nonsense projects, but very few people even question.

  9. Hi Dr. Hilgard, I'm not sure if you will see this at this point. I was a student in your social psychology class a few years ago. Your class was one of the most memorable, and gave me both an introduction to social psychology and, I believe more importantly, a more critical lens with which to view research. I have a strong desire to enter academia eventually (canine cognition/animal behavior, which combine my most loved and constant interests over the past 15 years), and while I do not want to join the field of social psychology myself, your class led me to a deeper appreciation and understanding for the work that goes into *good* science...and a better means of determining which results and studies might not be as great. Thank you for being the one professor to really make me aware of the replication crisis. Whether or not I actually end up in academia eventually myself, I will likely always love to read research and I appreciate that I had the opportunity to learn to be a better and more discerning consumer of research while at ISU. I'm sorry to hear that you're no longer teaching there, because your class had a pretty big impact for me, but best wishes on your new career path, I'm sure you'll make as much of an impact there. Thank you very, very much.

  10. Still reading this and admiring you in the second half of 2022.

    FYI this brought me here: Why Are Gamers So Much Better Than Scientists at Catching Fraud? https://archive.ph/Bxrou