Okay, in a very rare break for this blog, I’m going to write about someone that’s not theatre-related at all, or even arts-related. This is something that frequently crops up in arguments about the arts, but no more or less than anywhere else. But with an general election coming and the inevitable rise in poorly-researched claims to back up your favourite party, it’s about time I said something about this. I’m even going to be completely serious here are refrain from snarky asides that I usually make.
The practice I’m referring to is appealing to authority. This is where people attempt to back up their claims by citing the research of some sort of expert who is meant to back up the position. Now, in theory, this should be a good way of proving a point, showing someone else who knows what they’re doing and has done the research. But, in practice, this doesn’t stand up to scrutiny, and instead is used as a tactic to mislead people into believing their claim is proven when it isn’t. If you are doing it yourself, chances are you don’t realise you’re doing it because you are hearing what you want to hear and not thinking about whether this really supports you are strongly as I think it does.
So, as an attempt to help people appealing to authority to come up with decent arguments, and to enable the rest of us to spot the hogwash, I have taken inspiration from the legendary Graham’s Hierarchy of Disagreement. Using this model, I am pleased to introduce Chris’s Hierarchy of Appealing to Authority.
(Click on image for bigger version.)
As with Graham’s Hierarchy of disagreement, the triangular shape indicates that you can expect a lot more arguments of appeal to authority at the bottom of the scale than the top. It’s easy claim experts back their position without giving any detail – it’s a lot harder to claim why it’s backed or how well it’s backed.
Throughout this, I will be using a hypothetical example of someone wishing to argue that Bloggs’s widgets are more widgety that all other widgets. I may at some point give some real examples, but that’s a political minefield and I’m not ready to leave myself open to claims of taking sides. Bloggs’s widgets it is. What I hope this illustrates how easy it is to make it sound like your claim is back up by solid evidence, and how flimsy the evidence can actually be when put under scrutiny.
Anyway, here we go with the details.
AAH0: No substantiation at all
Asserts that the claim is supported up by authoritative sources with no evidence of what they are; only vagaries like “studies say”.
e.g. “Science shows Bloggs’s widgets are more widgety than all other widgets.”
Starting with the absolute worst kind of appealing to authority, it’s the one where people imply their claims are backed up by people who know what they’re doing without the slightest indication of who said it, let alone when this was said or where this can be checked. The language used varies, but it will normally be some variant of “researchers found” or “the science says”. Sometimes a newspaper article will use this in a title, but go on to give details in the text – that is fine. What we are talking about here, what gives this an AAH0, is the complete absence of substantiation accompanying this.
This is utterly worthless as an argument. If this unnamed study or unnamed researcher had a shred of credibility, they shouldn’t have any problem stating who said this and all open it to scrutiny. If they don’t say so when asked, it’s highly likely that they don’t believe it will stand up to cross-examination – or they could simply be telling outright lies that this exists at all. You can expect a multitude of well-rehearsed excuses when you ask for this (the current popular one I’m hearing is “It’s not my job to educate you”), but the bottom line is that as they’re the ones who said their claims are supported by authoritative sources, it up to them to provided the evidence. Until they do, you can safely dismiss this rhetoric out of hand. Remember, claims without evidence and be dismissed without evidence.
(Bonus point: A slightly better variant of refusing to provide any supporting evidence is “Google it”. This is a slightly better argument than “studies say …” for one reason and one reason only, which is that it implies that the information is easily available online. Very rarely, this carries some weight – if, for example, you tell a flat-earther to Google why the world is round, you can safely assume the top results will all say similar things. Most of the time, however, it’s a cowardly move, because you can debunk one source and they can simply say it wasn’t that one. Anyone who’s truly confident in the “Google it” remark should have no trouble saying which web pages you stand by.)
AAH1: Names an authoritative figure but no source
Names an authoritative figure (either an individual or an organisation) asserted to support the claim, but does not reference a verifiable source where the figure was supposed to have said this.
e.g. “Professor Henry Brubakker of the Institute for Studies says Bloggs’s widgets are more widgety than all other widgets.”
There are two common manifestations of this. One is where statistics are quoted supporting the claim and the source is quoted as an academic, or a government department, or a think tank, or something other individual or body who sounds like they know what they’re talking about. The other is where the claim is supported by a quote made by someone who’s meant to know about the subject. What all these tactics have in common is that only a figure in authority is cited, not where or when this was supposed to have been said.
This might sound a lot more credible than the worthless “studies say …” claim, but in reality these claims don’t carry much weight. There’s all sorts of problems with this, the big one being that that these claims are near-impossible to scrutinise. You have no way of knowing whether this was an authoritative statement backed up with evidence, or just an off-the-cuff remark that wasn’t meant to be taken as gospel at all. It might refer to something somebody once said in a conversation, meaning there’s no way to verify what was actually said. You can sometimes guess what source they are referring to, but it still leaves room for them to move the goalposts and say it wasn’t that one – and besides, it’s not your job to work out their argument for them. If they have a verifiable source, they can name it easily.
Very rarely, you can use an authoritative figure as credible evidence if that person is well-known for this view. If you cite Charles Darwin as your evidence for evolution, I can safely assume you’re referring to The Origin of the Species. But it’s really not that hard to say so yourself. Don’t be lazy.
AAH2: Cites a source without explanation
Asserts that the claim is supported by a specified source without indicating what information within the source is meant to back up the claim.
e.g. “In the Journal of Widgetology in 2006, Professor Henry Brubakker of the Institute for Studies says Bloggs’s widgets are more widgety than all other widgets.”
This is really the absolute minimum you should expect from someone appealing to authority. This time, as well as telling us who said it, we have a specific reference of where and when this was said. It could be a scientific paper, a newspaper article, a recorded speech at a conference, a government statistical publication, or many other things; what’s important is that there is someone you can go and check to see whether it actually backs up their claims. But the shortcoming is that the source is cited without any explanation of where it backs up the claim.
The problem is that without an explanation, it is too easy to cite a source that looks like it backs up the claim at first glance but doesn’t on closer inspection. Often it will merely cover the general area under discussion, so an outlandish claim that an alternative medicine cures arthritis might link to a wordy article about arthritis in general. And a common tactic used is that when challenged about how this actually backs up their claims, they will blame you for not having read the article properly. Do not accept that excuse. If they’re so sure this source backs up their claim, they can say where. A similar tactic is to use a heavily technical source in the hope that no-one will understand it well enough to realise it doesn’t support the claim at all.
A context-free citation needs treating with suspicion. If they won’t explain where in the source their claim is backed up, there’s a high chance it’s because the source back up their claim either badly or not at all.
AAH3: Cites information within a source
Cites both a source and specific information within the source supports the claim, but takes information at face value and does not explain how it was worked out.
e.g. “In the Journal of Widgetology in 2006, Professor Henry Brubakker of the Institute for Studies says Bloggs’s widgets are 42% more widgety than all other widgets, as shown in table 4.2 of the paper.”
Moving up the credibility scale a bit further, this addresses the problem above by specifying not only a source, but also where in the source this information is backed up. It could be a chart or a table, or it could be a specific quote of text, but whatever it is, it has to be the specific source that backs up the claim. (Very occasionally, you can accept a source only as an AAH3 if it’s obvious where this is backing up the claim. If you cite a book called “Why smoking causes cancer” as your evidence that experts believe smoking causes cancer, we can safely assume that the book does indeed say this; similarly, if you use a government statistical bulletin to say that unemployment is 5%, we can safely assume you are referring to the headline figure of that source.) What distinguishes this from AAH2 is that it is not left to other people to search through the source and find out where it supports the claim.
But people who think that this is a killer argument – even one from a respected scientific journal – are misguided over how this works. Anything shown in isolation, be it a table, quote or chart, has a weakness that it doesn’t necessarily mean what you think it means. Often, figures shown in charts or tables are only incidental to the subject under discussion and therefore aren’t reliable to consider in isolation. Or there might be caveats or margins of error so large that the data isn’t precise enough to prove the point being made. Or the methodology used to obtain this could be so flawed that the figures or table are worthless. The information can of course be checked, and it may turn out to be credible, precise, and sound, but the point it it’s you who’s checking this, not them. Anyone who stops reading as soon as they find a quote, chart of table that suits them needs to be treated with suspicion.
You can get away with this if you are confident that the source you are quoting will stand up to scrutiny. But you are a lot better off checking and understanding the source yourself before other people try to debunk it. If there’s any flaws, you want to discover them before your argue this backs up your point, not after.
(Bonus point: It helps if the source being quote is a reliable one. Of course, it’s vague as to what constitutes a reliable one. Broadsheet papers are generally more reliable than tabloids, websites with editors tend to be more reliable than self-published blogs, but as a rule of the thumb, the publication should have something to lose if it is discredited. The gold standard is generally regarded to be peer-reviewed scientific journals. However, none of these guarantee the information is correct. Peer-review is only intended to check there’s nothing obviously wrong with the paper – it’s still open to other academics to dispute the details or even the conclusion. So whilst a reliable source is stronger than an unreliable one, there’s no substitute for scrutinising it yourself.)
AAH4: Understands a single cited source
Cites both a source and the information within the source that supports the claim, and explains how the source arrived at this information.
e.g. “In the Journal of Widgetology in 2006, Professor Henry Brubakker of the Institute for Studies conducted a trial of widgets from different suppliers, where widgets were selected and random and assessed for widgetiness using the insitute’s widgetometer. From this it was found that Bloggs’s widgets are 42% more widgety than all other widgets, as shown in table 4.2 of the paper.”
Now we reach the level were appealing to authority starts to get my respect. So far, all the levels described are, at best, boil down “I am right because this expert says so.” At this level, this isn’t good enough any more. Now you must explain why they said so. Not only must you cite an authoritative source and say where in the source your claim is supported, you must also understand how it came to that. You don’t need to understand every last detail (although it helps), but you do at least need to understand the basics. A person who cites the unemployment figures who understands who counts as unemployed and how these figures are worked out (hint: no, it’s got nothing to do with who’s on Jobseekers’ Allowance) will qualify for AAH4. A person who cites the same figure without understanding this does not.
This is the point of the scale where it can become difficult for novices to compete. Without understanding the subject in question, it becomes difficult to explain how the evidence in your chosen source was worked out. It varies from subject to subject: it is near-impossible to understand a maths paper if you’re not a degree-level mathematician; other subjects you might stand a better chance if teach yourself the basics. Sometimes this can mean a person who is not familiar with the field under discussion does not stand a fair chance in an argument with someone who is, but that’s tough luck. Crying foul over your opponents’ levels of knowledge does not invalidate their arguments or evidence.
The weakness of citing a source this way is, of course, that it’s only one source. In isolation, there is no way of knowing whether the source is one of many sources in widespread consensus, or a lone voice surrounded by a mass of conflicting sources. No matter how well a source is understood, there could still be something fundamentally wrong with it, and without knowing about other sources that might pick up on these problems, you will be none the wiser. To strengthen the case beyond this, you start to need to look for multiple sources.
(Bonus point: you have a stronger case if you check not only how the source backs up your claim, but also its limitations. It is very rare that a cited source will back up a claim perfectly – it may cover something similar but different to what you’re claiming; there might be margins of error to take into account; there could be caveats within the source; or there may even be weaknesses in the method that you yourself pick up on. Acknowledging these limitations is not a sign of weakness, it’s a sign that you study sources before decided if they support your position instead of finding sources that appear to support your position and ignoring all details that don’t suit you. But this does not compensate for the weakness that it’s only one source.)
AAH5: Understands multiple independent sources
Cites multiple sources that are independent of each other, showing how which information within the source support the claim and how these sources arrived at this information.
e.g. “In the Journal of Widgetology in 2006, Professor Henry Brubakker of the Institute for Studies conducted a trial of widgets from different suppliers, where widgets were selected and random and assessed for widgetiness using the insitute’s widgetometer. From this it was found that Bloggs’s widgets are 42% more widgety than all other widgets, as shown in table 4.2 of the paper. Other studies using different methods that I am happy to explain have come to similar conclusions.”
The obvious way to overcome the limitations of one source is, of course, to provide more than one source. Note, however, there are two important conditions to this.
Firstly, all the conditions that apply to AAH4 apply to AAH5 too. In order for multiple independent sources to carry any real weight, they have to be properly understood. Anyone can list a whole load of sources that allegedly prove the point they’re trying to make, but without demonstrating and understanding of how this was worked out, it doesn’t mean much. The argument of “I am right because this expert says so,” carries little more weight if is becomes “I am right because these experts say so.” All it proves is that they’re good at finding sources that back up what they already believe. That doesn’t automatically mean they’ve been cherry-picking and ignoring all the sources that don’t suit them, but you can be suspicious they did.
Secondly, the sources have to be independent of each other. Imagine somebody who gets challenged for measuring a distance with a bent ruler; so that person measures the distance again with the same ruler and says this shows the distance is correct. Sounds absurd? Well the equivalent fallacy here is where a researcher (particularly a researcher whose findings are disputed) does the same research over a wider sample, get the same result, and that is taken as proof of the original findings. It is not. If the original method was flawed, repeating the experiment with the same method will probably have the same flaw. If the original flaw was undetected, repeated the experiment will probably not bring the flaw to light. Beware also of other researchers who copy the same method used by the original source. At best, it doesn’t address the possibility that the original method is flawed; at worst, it’s a cynical ploy to use a method known to give them the results they want, not caring about whether the method is reliable. The only time you should less this pass is if there’s widespread consensus that the method used is valid and reliable in these circumstances. But ideally you want different researchers to come up with the same results using different methods, of – if they must use the same method – to choose to use it independently of each other.
But if these two conditions are met – if the multiple sources are both properly understood and the sources are truly independent of each other – you have a very strong case to back up your claim. There’s just one thing that can make it stronger.
AAH6: Proves academic consensus
Shows that when all authoritative sources are taken into consideration – including sources that don’t support this position – they collectively back the claim.
e.g. “There have been 24 studies in peer-reviewed journals comparing widgetiness of Bloggs’s widgets to other widgets, of which 21 concluded the Blogg’s widgets are more widgety than other widgets, 2 said they were about the same and 1 said they were less. A typical example is in the Journal of Widgetology in 2006, Professor Henry Brubakker of the Institute for Studies conducted a trial of widgets from different suppliers, where widgets were selected and random and assessed for widgetiness using the insitute’s widgetometer. From this it was found that Bloggs’s widgets are 42% more widgety than all other widgets, as shown in table 4.2 of the paper. Other studies using different methods that I am happy to explain have come to similar conclusions.”
There is just one thing that AAH5 doesn’t do. It finds multiple independent sources that back up the claim, but it does not take into account whether there’s any sources that don’t back up the claim. You might have found five sources that support your position, but what if there’s twenty-five that don’t? So the one thing you can do to top this is to take into account all the sources.
It’s fair to say that this level of appealing to authority is realistically only achievable by people who are already experts in the field in their own right. It’s easy to find a source supporting a claim, harder to understand it, but the hardest thing to do is know if you’ve covered all the relevant sources. There are ways of doing this – you can read review papers that aim to consider all the research out there, you can search for keywords in search engines for academic papers (not Google, it doesn’t have that level of access), you can ask academics in the field what their understanding is – but without a good understanding of the subject, or at the very least a good understanding of how academic research works, in general, you have to be aware the cards are stacked against you here.
But if you can do it, you have pretty much the strongest possible position to back up your claim. Nothing is ever 100% certain, but there’s no higher level of experts you can use to overturn this. The only way you can beat it is to take on the research itself and come up with an extremely good argument as to how everyone else is getting it wrong. It can be done – after all, Charles Darwin overturned the academic consensus that the world was created 6,000 years ago – but most of the time anyone who tried this can expect to be mauled by the experts mercilessly. So if you are able to prove academic consensus, well done, you have almost certainly won the argument.
What it means
Now we’ve listed all of these, an easy misconception would be that whoever comes up with the highest level wins the argument. Not necessarily.
The most important caveat is that an argument that appeals to authority is only as good as the source used. At the risk of pointing out the obvious, even the most authoritative-looking sources can be proven wrong. Someone might back up their claim with a research paper from a credible source, showing where this backs up the claim and demonstrating a thorough understanding of how this was worked out, and this would be an AAH4. But just because a paper is properly understood doesn’t guarantee it’s right. If someone else finds out what’s wrong with it and thoroughly debunks it, this source may suddenly carry no more weight than an AAH0 – in other words, the discredited paper is just as worthless as phrases such as “educate yourself”.
Conversely, any level of appeal to authority is open to the strengthened to a higher level later. If someone backs up an argument with a source without explanation, that’s an AAH2, but if someone else challenges this and the original person comes back to expand on this, this could go up to AAH4 – maybe even AAH5 or AAH6 if the person looks at other sources too. You can’t dismiss someone’s source just because they didn’t provide all the details the first time round. By all means dismiss it if they refuse to give details after you challenge them, but if they do provide details, they must be considered fairly.
Ultimately, there’s just two things this is meant to achieve. Firstly, I hope this will encourage people who resort to appealing to authority to stop and think if their chosen experts actually are backing up their claims. Hopefully go and check the if the sources they are counting on actually do support this position. And ideally, if you realise these sources say something different, maybe change your mind in light of the new evidence.
But secondly, I hope this will provide people with the confidence they need to challenge and dismiss baseless claims. It’s easy to mislead people into believing the authoritative sources prove points that aren’t really proven, but it’s also easy to spot this and challenge it once you know what you’re looking for. At more people need to do this.
Ultimately, if you are going to use someone else’s research to back up your claims, the onus is on you to show why it does. Expecting this to put your claim above criticism until other people have read your source from beginning to end doesn’t cut it. And if you’re the person on the receiving end, you are completely within your rights to refuse that sort of citation as evidence. Don’t be afraid to do so.