Great article again. What do you think of using expert consensus as a guide to beliefs? As you say, you can't fully and responsibly critically think everything, since you don't have the time and energy.
I'm not an expert on climate change but given the strong expert consensus that a substantial portion of warming is due to human activity, I accept that conclusion. I'm not an expert on vaccination, but given the strong expert consensus on how well they work, I accept that conclusion. I'm not an expert in biology, but given the strong expert consensus that evolution explains the origin of species, I accept that conclusion. Etc. I always want to take a cursory glance at WHY a particular position is the expert consensus (i.e. what evidence they're relying on), but in the end, some trust in the experts is inevitable.
Where you draw the line for "strong expert consensus" is debatable but I feel like some acceptance of the principle above is important for avoiding a complete rejection of valuable institutions and emboldening of crazy internet people.
Thank you Erman and difficult but excellent question! I’m not entirely sure where I fall on this yet. Here are some of my thoughts on it.
I think there is nothing wrong with relying on “strong expert consensus” as long as you are transparent about it and are aware of the limitations. By being transparent I mean that if someone asks you about your stance on climate change that you respond by indicating that you are merely relying on expert consensus. That this isn’t something you have spent a lot of time thinking about. I think this matters because at the end of the day you didn’t come to this conclusion, the experts did.
I think it’s also important to be aware of the limitations of expert consensus. Just because experts largely agree doesn’t mean that it’s true. Also to consider what is actually in the tin of “expert consensus.” That this isn’t always a situation where 100s of experts are independently researching something and coming to the same conclusion. I expect it’s more often a handful of prominent experts claiming something with most people falling in line and then spreading the word. I can only comment on psychology but sometimes some leader in a field publishes a single paper in a high impact journal and it becomes “expert consensus.” Thinking, Fast and Slow by Kahneman is still being uncritically recommended by psychology professors years after Kahneman has distanced himself from some of the research he refers to in the book.
Also I know from my own field that it can also happen that there are a lot of people that appear to be experts and that there are a small subgroup of true experts. With the apparent and true experts in disagreement. Sometimes it may seem as if there is broad expert consensus, but you don’t know if there is valid critique coming from subgroups.
It's also important to consider that once something becomes expert consensus that it’s very difficult to overturn/update it. This is Max Planck’s idea of science progressing “one funeral at a time.” So, there might be something better out there, but it’s being dismissed due to not being in line with expert consensus.
You already addressed the importance of knowing what evidence the experts are relying on. I think there are two main reasons for this.
1. The experts’ threshold for evidence may be different to yours. Experts are aware of how difficult it is it conduct studies that yield strong evidence. So, an expert’s threshold for evidence might actually be lower than yours. I’m not precisely sure about the details of the following example because it’s been a while since I’ve read into it, but lots of the early evidence of cannabis and LSD causing psychosis is pretty flimsily. Correlational designs and lots of clinical anecdotes and yet it is/was expert consensus (not exactly sure what the current thoughts on it are). Once I looked at the evidence myself I was pretty appalled at what the expert consensus was based on.
2. We rarely get the “expert opinion” from the experts themselves. We usually don’t hear the experts discuss the evidence in a nuanced way. Instead we usually hear the “expert opinion” from other people. People that are usually louder and less nuanced. Now that I think about it, I don’t think I’ve ever heard from an expert on climate change, but I’ve heard everyone and their mother talk about how important climate change is.
I think that an idea that nicely ties all of this together is thinking in probabilities which is something I purposefully left out of this essay because I am only in very beginning of adopting it. Expert consensus doesn’t mean something is 100% true, but it does substantially increase the chances of it being true. Let’s say whatever you want to call expert consensus is on average true 85% of the time. Those are good betting odds, but it’s not 100%. If you want more certainty than that it’s our job to look past the label and look into the tin of “expert consensus.”
(On a practical note, this isn’t how I operate day-to-day. I also trust the expert consensus on the three topics you’ve mentioned and I have never examined the evidence myself.)
Thanks for the detailed thoughts - definitely agree on the major points. In my experience, most scientists these days are reasonably skeptical about single papers published in high impact journals, so that unreliable route to expert consensus might happen less often that you expect (though I'm also familiar with the kinds of psychology professors you're talking about).
But yes, my main concern is the more practical one you end with. We have to rely on shortcuts of some form to make big decisions in life (e.g. Do I trust vaccines?), and I believe the best available shortcut is expert consensus. If we accept that gets you to 85% truth, my assumption is the alternative shortcuts (e.g. personal faith, a favorite expert, a random expert on Joe Rogan's podcast, etc) will get you much lower. It's the same reason I try to get multiple estimates/opinions from expert plumbers when a pipe leaks—I trust they understand my plumbing better than most but don't want to run the risk of trusting a single outlier.
I enjoyed reading this! A good reminder to trust yourself as well as others ... balance :)
Thanks Sarah-Anne! I agree but sadly easier said than done :)
Great article again. What do you think of using expert consensus as a guide to beliefs? As you say, you can't fully and responsibly critically think everything, since you don't have the time and energy.
I'm not an expert on climate change but given the strong expert consensus that a substantial portion of warming is due to human activity, I accept that conclusion. I'm not an expert on vaccination, but given the strong expert consensus on how well they work, I accept that conclusion. I'm not an expert in biology, but given the strong expert consensus that evolution explains the origin of species, I accept that conclusion. Etc. I always want to take a cursory glance at WHY a particular position is the expert consensus (i.e. what evidence they're relying on), but in the end, some trust in the experts is inevitable.
Where you draw the line for "strong expert consensus" is debatable but I feel like some acceptance of the principle above is important for avoiding a complete rejection of valuable institutions and emboldening of crazy internet people.
Thank you Erman and difficult but excellent question! I’m not entirely sure where I fall on this yet. Here are some of my thoughts on it.
I think there is nothing wrong with relying on “strong expert consensus” as long as you are transparent about it and are aware of the limitations. By being transparent I mean that if someone asks you about your stance on climate change that you respond by indicating that you are merely relying on expert consensus. That this isn’t something you have spent a lot of time thinking about. I think this matters because at the end of the day you didn’t come to this conclusion, the experts did.
I think it’s also important to be aware of the limitations of expert consensus. Just because experts largely agree doesn’t mean that it’s true. Also to consider what is actually in the tin of “expert consensus.” That this isn’t always a situation where 100s of experts are independently researching something and coming to the same conclusion. I expect it’s more often a handful of prominent experts claiming something with most people falling in line and then spreading the word. I can only comment on psychology but sometimes some leader in a field publishes a single paper in a high impact journal and it becomes “expert consensus.” Thinking, Fast and Slow by Kahneman is still being uncritically recommended by psychology professors years after Kahneman has distanced himself from some of the research he refers to in the book.
Also I know from my own field that it can also happen that there are a lot of people that appear to be experts and that there are a small subgroup of true experts. With the apparent and true experts in disagreement. Sometimes it may seem as if there is broad expert consensus, but you don’t know if there is valid critique coming from subgroups.
It's also important to consider that once something becomes expert consensus that it’s very difficult to overturn/update it. This is Max Planck’s idea of science progressing “one funeral at a time.” So, there might be something better out there, but it’s being dismissed due to not being in line with expert consensus.
You already addressed the importance of knowing what evidence the experts are relying on. I think there are two main reasons for this.
1. The experts’ threshold for evidence may be different to yours. Experts are aware of how difficult it is it conduct studies that yield strong evidence. So, an expert’s threshold for evidence might actually be lower than yours. I’m not precisely sure about the details of the following example because it’s been a while since I’ve read into it, but lots of the early evidence of cannabis and LSD causing psychosis is pretty flimsily. Correlational designs and lots of clinical anecdotes and yet it is/was expert consensus (not exactly sure what the current thoughts on it are). Once I looked at the evidence myself I was pretty appalled at what the expert consensus was based on.
2. We rarely get the “expert opinion” from the experts themselves. We usually don’t hear the experts discuss the evidence in a nuanced way. Instead we usually hear the “expert opinion” from other people. People that are usually louder and less nuanced. Now that I think about it, I don’t think I’ve ever heard from an expert on climate change, but I’ve heard everyone and their mother talk about how important climate change is.
I think that an idea that nicely ties all of this together is thinking in probabilities which is something I purposefully left out of this essay because I am only in very beginning of adopting it. Expert consensus doesn’t mean something is 100% true, but it does substantially increase the chances of it being true. Let’s say whatever you want to call expert consensus is on average true 85% of the time. Those are good betting odds, but it’s not 100%. If you want more certainty than that it’s our job to look past the label and look into the tin of “expert consensus.”
(On a practical note, this isn’t how I operate day-to-day. I also trust the expert consensus on the three topics you’ve mentioned and I have never examined the evidence myself.)
Thanks for the detailed thoughts - definitely agree on the major points. In my experience, most scientists these days are reasonably skeptical about single papers published in high impact journals, so that unreliable route to expert consensus might happen less often that you expect (though I'm also familiar with the kinds of psychology professors you're talking about).
But yes, my main concern is the more practical one you end with. We have to rely on shortcuts of some form to make big decisions in life (e.g. Do I trust vaccines?), and I believe the best available shortcut is expert consensus. If we accept that gets you to 85% truth, my assumption is the alternative shortcuts (e.g. personal faith, a favorite expert, a random expert on Joe Rogan's podcast, etc) will get you much lower. It's the same reason I try to get multiple estimates/opinions from expert plumbers when a pipe leaks—I trust they understand my plumbing better than most but don't want to run the risk of trusting a single outlier.
Agreed! On average I'd prefer to go with expert consensus rather than an individual expert's take.
This is great. I think Carl Sagan's Baloney Detection Kit is the best "in your pocket" tool for critical thinking. https://centerforinquiry.org/learning-resources/carl-sagans-baloney-detection-kit/
Thanks for your comment Thomas! I fully agree.