Critical Thinking and Evaluating Information

2007 October 24

By Matt

One of the common things I hear from people deals with the evaluation of information. With so much conflicting data out there, so many self-proclaimed experts, how can you ever be sure what is right and what isn’t?

Now, this article is going to get a bit in-depth. If you’re new to logical thinking and critical analysis, it might go over your head. But I’ll try to be user-friendly.

Much of this deception happens because most people aren’t equipped with the right tools to evaluate information in a critical fashion.

Science and its related disciplines are focused on the scientific method, which is a strict means of observing phenomena, gathering data, and interpreting it. Science is based on the idea of fallibility; if an idea cannot be challenged and subject to review, then it is not a scientific theory.

We can evaluate training and nutritional concepts in much the same way. Any statement can be subject to challenge; in fact, this is how it should be. A strong claim will meet the criticism, and thus display that it is valid. A weak claim will either be improved, or shown to be inadequate (ie, bullshit).

A logical argument is constructed from premises which then support a conclusion. Logical fallacies are dishonest, misleading, or incorrect construction of arguments, and there is in fact an entire field devoted to studying this process.

Suffice it to say, the reasoning is the most important part of evaluating any claim. Strong claims will be backed with evidence and solid reasoning.

Example of a strong claim:

“The sun is bright.”

This is a strong claim, for obvious reasons. Approximately 50% of any given day in most places on the Earth can prove this is true without a doubt. Even those unable to observe the sun, such as the blind, could be convinced of this point beyond a doubt with reasonable education in physics and indirect observation (such as the feel of intense sunlight on the skin, or knowledge of Earth’s biosphere), which would be examples of sound logical reasoning.

“Lifting weights will make your muscles bigger.”

Again, this is a strong claim. It can be supported by empirical evidence; just go to any gym and compare the average person to the average person on the street. Although this is a weak argument, it is still a supporting premise of the claim.

Stronger evidence comes in the form of academic research, which has not only confirmed that weight training increases muscle mass at the macro level, but has shown that there are physiological and chemical responses to the activity. As opposed to the above anecdote, academic research is much harder to dismiss out of hand. While a smart person could dissect the anecdote, peer-reviewed research isn’t so easy to pick apart, especially when the body of research is relatively wide.

While any piece of data in isolation may not be strong, when the entire body of data is more or less established we can be fairly certain of the conclusions.

A weak claim on the other hand will either have no evidence (ie, pulled out of someone’s ass), or it will be based on poor reasoning, as in cases where there is solid evidence but poor interpretation by the person making the claim.

Example of a weak claim:

“The sky is red.”

This one is easy. You can walk outside on most any day and see that the sky is not in fact red (we’ll excuse for a moment the “city lights” phenomenon that can cause a reddish-pink coloration to the night sky). But what about subjective interpretation (the hippie stoner existential argument)? You know, just because you think the sky is blue doesn’t mean everybody else perceives it that way, right?

Wrong. Guess what colors really are? They’re wavelengths of visible light. Wavelengths can be measured objectively with rather simple equipment.

So now we’ve got empirical evidence, both in the form of a consensus (most people would not deny the claim that the daytime sky is in fact blue), and in objective, repeatable observation (light wavelengths).

This would be an example of blatant bullshit. But there can be claims that are harder to see through. Another such approach involves the way research and data are interpreted, as opposed to the argument itself being flawed.

The argument itself can be sound. But if it’s based on a false premise, ie the research doesn’t say what it is claimed, this is a failure of interpretation, and is a form of intellectual dishonesty. This doesn’t mean the person is lying outright. It means that they are being deceptive by using poor reasoning. This would be an example of cherry-picking, in a variation of the “stolen concept” fallacy.

All poor reasoning can ultimately be reduced down to the fallacies of logic, and this includes both the argument itself and the interpretation of even valid data.

In the fitness industry, we inevitably have to frame things in the context of deductive reasoning. This means simply that any theory or claim must be based on what we have observed to happen. Due to the nature of research, which includes everything from inability to study certain phenomena to ethical considerations, we unfortunately do not have a complete body of academic research to draw upon. This means that we must therefore draw on empirical observations, such as the experiences of coaches and trainers. Even anecdotes can take on a form of validity in the absence of other evidence.

Strictly speaking, this is inductive reasoning, in which premises are assumed to support a position, but due to lack of evidence they do not confirm it. Obviously, inductive reasoning is a weaker form of argument than deductive reasoning. However, inductive reasoning is the only tool we have in the absence of hard data.

What this means is that it we can end up having claims made that are impossible to categorically disprove. In science, claims tend to be disproven by new data. Once upon a time it was thought that the sun might be a giant fire in the sky. This was disproven with improved knowledge of physics and observation of stellar phenomena that made it far more likely the sun was a giant fusion reaction.

In the fitness industry, we have a similar dilemma. Let’s say that Trainer X says that he did standing barbell curls for 12 sets of 8 reps, and his arms are 19″, so therefore you can get 19″ arms by doing standing barbell curls for 12 sets of 8 reps. How do you disprove that?

The simple answer is, you don’t. You can’t really disprove claims, per se. However we still have tools to analyze such claims, and therefore we can call it a bad argument even if we can’t outright disprove it.

1) Cause and effect. How does Trainer X know that standing barbell curls for 12 sets of 8 are what resulted in his 19″ biceps? Can he produce data that conclusively show that his 19″ arms were caused by his protocol?

2) Burden of proof. How can he say for sure that it will work for any of his clients? Has he tested it with a random sampling and been able to show that it is repeatable?

3) Applicability. Why should we assume that 12 sets of 8 will build 19″ biceps, when we have numerous examples of guys doing just that and experiencing no growth at all? Why is there a disconnect between what we see, ie tons of guys “blasting their biceps” and getting nowhere, versus what Trainer X is claiming?

A better argument might be “Training the biceps with adequate volume and adequate loads can result in larger arms under the right conditions.” Even that still has its flaws, but at least it 1) can show a link between the cause and effect, 2) can be tested and repeated with near 100% reliability in the real world, and 3) is applicable to most of the relevant population (ie, people lifting weights in the gym).

In short, we analyze the claims in context of data we do have, and put them through a rigorous examination of the reasoning. If they pass the muster, even though we don’t have data supporting the exact claim, we can give it credence. Until something comes along that might show otherwise. If they don’t stand up to the criticisms, Trainer X might want to have a long think about why.

Trainer X’s advice to get 19″ arms might not be valid for everybody, but by the converse, that doesn’t mean you can tell Trainer X that it isn’t effective for him, either. It might well be effective for him, even if it isn’t advisable for general advice. This is a subtle distinction.

To go off on a tangent, that’s exactly why we end up with so much fluff and dogma in the fitness industry. The personalities in the field tend to treat it as a religion rather than a scientific process. All the little techniques and programs with clever acronyms become a faith rather than a process of refinement.

The ideal approach is to use a rational, methodical approach to examine what you do, and always be on the lookout for ways you can improve.

If you learn to do things better, then you’ve just improved your ability. Religion is static; science is continually improving. Most people in the fitness industry do not think like this. It would be to your great advantage to do so.

Recently I had a friendly debate with a gentleman that feels that overhead pressing is a risky endeavor. I pointed out to him that for years upon years, people overhead pressed just fine; from Eugen Sandow and his contemporaries in the late 1800s (and probably before that) right on up until the 1960s when the Press was dropped from Olympic weightlifting competition. In fact, it wasn’t until the explosion of the bench press as “the” exercise in gyms that shoulder injuries started becoming common. Added to that, I’ve known or heard from quite a few people that almost exclusively OHP and have zero shoulder problems, which include former benchers that have had shoulder issues.

Now, this was never meant as anything more than a real-life anecdote that could call his conclusion into doubt. As anecdote, it was quite open to being disproven by research that showed a causal relationship between overhead pressing and injury, with the impact of the bench press controlled for. It’s pure inductive reasoning, not a 100% conclusive proof of the argument.

Admittedly, even said anecdote had weakness: there was no controlled data to show that the old-timers and OLers did not have a preponderance of shoulder injuries, either. Certainly not iron-clad evidence by any means.

But! We have to remember the burden of proof: The claim is “overhead pressing leads to injury”, not “the old-timers didn’t have shoulder injuries”. That, and anecdote in and of itself is not an invalid argument: it only becomes so when used in an illogical manner.

Think back to the big picture: if guys were really wrecking their shoulders to that degree, don’t you think it’s reasonable to assume that it would have been mentioned somewhere? Strangely, it isn’t. No this isn’t proof that such things aren’t happening, but it is compelling reason to question any claims to the contrary.

Well, I got blasted for bringing that up, accused of using bro logic to support my argument (Note: this is actually fallacious in itself, since I presented no argument besides “anecdote never showed this” to begin with, but I digress).

Now, what you’d expect for the claim “overhead pressing leads to injury” to be strong is 1) research showing a causal link 2) at least a correlation to the real world happening, and 3) at least a reasonable argument that could convince one that there was a risk.

The anecdote only calls 2) into question, and even then doesn’t disprove it per se. However, in the absence of 1) and 3), it’s pretty damning.

When said individual was questioned for data, he did indeed produce. However (and unsurprisingly) it wasn’t quite what he thought, which brings up another point: extrapolation and interpretation of data.

The entire contention dealt with the acromion process, which is a component of the scapula that articulates with the clavicle to create the acromioclavicular (AC) joint at the top of the shoulder. According to the literature, there are three “types” of acromion; the entire danger from overhead pressing was a result of the alleged type III acromion that would cause cumulative damage to the bursa and tendons of the rotator cuff, due to impingement when the arms were lifted over head.

The research that was presented did indeed confirm that this was a possibility. However, there was more to it. This same research also showed that most researchers weren’t even convinced that the type III acromion even existed as anything but a degenerative condition. This was further supported by information showing that the type III acromion had only ever been observed as a degenerative condition in aged individuals.

The argument against overhead pressing was that, statistically, X% of people had the type III acromion and thus were prone to shoulder injury from lifting overhead. The research presented unfortunately did not show this; since the type III acromion is in fact only found in specific populations, it means that even in the worst case, you can be reasonably certain of who is at risk. Secondly, it was intellectually dishonest to assume that the statistic applied to all populations across the board, when that clearly was not the case.

This is an instance of cherry picking: picking out statements that support your argument and ignoring the rest, which is similar to the so-called stolen concept and a reversal of the strawman fallacies. You use the data that suits your needs while ignoring the overall argument.

There were other attempts at distraction, as well: style over substance arguments (you’re a dick, so you’re wrong), and a lovely red herring regarding “proper movement patterns”, which I love, but that’s a whole different rant. Suffice to say, I have two fundamental problems with this: firstly, if you’re doing your job as a coach, the athlete will already be moving properly; and secondly, the body is not this fragile.

That’s an argument that HIT has been using for decades to justify its anti-explosive, pro-machine ideology, and they’ve never managed to do anything but hide behind logical fallacies either.

If you’re that worried about “poor movement patterns” in the gym, just what the hell do you think is going to happen on the field, the track, or the octagon? Really now…if we’re going to talk injury risk, let’s talk injury risk, and not throw in a red herring to distract from the fact that you just don’t know what you’re doing. That goes for overhead pressing, Olympic lifts, and any other damn thing you want to label “bad”. Neverminding that this entire tangent has very little to do with the original argument that overhead pressing is inherently bad due to shoulder anatomy.

Then we finished up with an appeal to consequence, the logical version of Pascal’s wager: you might end up hurt, therefore you can’t take the risk of doing such an exercise. Well, that’s nice and all, but it’s not support for the conclusion.

So to sum up my opponent’s argument: There was no data showing that there was any correlation, let alone a causal link, between overhead pressing and shoulder injuries; this rules out 1) from above. In light of this, the anecdote wins out: the research is not enough to overcome the assertion, which knocks out condition 2). The stolen concept/cherry-picking misrepresentation knocks out 3), and thus serves as the final nail in the coffin. Add some fallacies of distraction and moving the goal posts, and you’ve got a textbook example of “bad claim”.

Here’s what’s even better though, and might boggle some of you: Even now I cannot concede that I am right, due to the limits of the inductive process. I can only say that his argument failed to meet the requirements for proving his claim. That’s what is wonderful about this process: if a study comes out tomorrow that proves without a doubt that there is a causal link between overhead pressing and shoulder injuries, then I will gladly concede the point and alter my training methodologies accordingly. I don’t expect this to happen, based on precedent, but it is certainly a possibility.

Ultimately it boils down to the religious fervor I discussed above. People become so attached to an idea, so personally invested in it, that they cannot put it up for criticism, nor accept it when their pet idea is shot down. Even research, and those that supposedly have higher education, are not above reproach. Circular logic, whereby the premise is accepted as valid, and appeals to authority, where status and credentials are considered evidence, are considered the norm in that realm, but researchers and advanced degrees are not infallible. Even they should be questioned and held to scrutiny.

It would be better to take on an “outcome-based” outlook, whereby the particular approach is irrelevant so long as it gets the job done.

Most don’t do that though, as they revert to a faith-based model.

Coaches, trainers, and even individual gym-rats that can learn to think with the results-based model in mind will fare far better than those individuals that cannot get past their own myopia and dogma.

I’ve said more than once that being a critical and analytical thinker is far more important in this field than anything else. You need to be able to see not only what works, but be able to filter through the substantial bullshit. Having that skill alone will put you miles ahead of your colleagues. Taking a few classes on logic, research methods, and statistics would, sadly, end up taking a new trainer or coach a lot farther than any education on exercise science in this current environment. The information is there; you simply need to be able to cut through the chaff to get to the goodies.

There’s a few simple rules of thumb I keep in mind when I’m dealing with a dogmatic know-it-all:

1. Question claims. Ask for data. If something sounds like bullshit, it probably is, especially if you’re judging it in the context of the known data. On the other hand, don’t be closed to the possibility: you might just be surprised. In any event, scrutinizing the claim is the only way to find out. Examine the logic; look at the data. If it fits, you might be on to something. If not, ask more questions, because you just might be dealing with a bullshitter.

2. Be aware of logical fallacies in arguments, both in the reasoning and in the interpretation of evidence.

Understanding the handful of most common fallacies, many of which I’ve lain out here, will go a long way towards this.

3. Think of the big picture. Nothing happens in a vacuum. This means framing things in the context of empirical evidence as well as academic research. Like it or not, the guys “in the trenches” are a very big source of our information on coaching. Yes, this is inductive reasoning, but in the absence of other valid evidence, it’s all we have to go on sometimes. This doesn’t make it invalid, no matter how much somebody wants to kick and scream. It only becomes invalid if research or even common-sense reasoning can show it to be flawed.

Further, if something in the lab contradicts something in the “field”, ask why. Don’t just latch on the the research; question why there’s a contradiction. If you actually look at things, you might find that a seeming paradox will simply vanish.

4. Remember that this is not an exact science. Much of training and coaching is still an artform, and as such there are not always hard right or wrong answers. Anyone that’s making bold claims should be held up to bold scrutiny.

Comments are closed, but you can still come ask a question on the forum.