Library Skip to Main Content
It looks like you're using Internet Explorer 11 or older. This website works best with modern browsers such as the latest versions of Chrome, Firefox, Safari, and Edge. If you continue with this browser, you may see unexpected results.

Research Skills Tutorial

Validity

Validity asks the question, "How do we know what we know?" Every field of study answers that question differently, but there are some ideas that are generally considered invalid, no matter whether an information source is scholarly or non-scholarly, and no matter what discipline or subject area it falls under.

In an earlier section, we talked about how the intended audience and purpose of an information source can slant the information in it. The concepts of ideology, agenda and bias are related to that. 

  • Ideology is a belief system shared by a group of people. Religions, political groups, and advocacy groups have ideologies. Not all ideologies are bad. And just because an author subscribes to an ideology does not mean that the information source necessarily has an ideological agenda. 
  • Agenda is a set of goals shaped by an ideology. If an information source has an agenda, that means it is not strictly informational or educational. It is probably actually a persuasive information source or a piece of propaganda, even if it is pretending to be otherwise.
  • Bias is the tendency of an information source to selectively over-emphasize some things and de-emphasize other things in such a way that it unfairly favors a certain conclusion or point of view.

A credible information source will not try to tell you how to feel about the information.

Humanities

In the humanities, the author is either constructing a worldview, or (more often) adding to, refining, and correcting a worldview that other scholars have created. That worldview needs to be self-consistent and consistent with the evidence. It needs to be able to support new discoveries and insights. And it often needs to be beautiful. 

Here are some criteria for validity that can be generalized in the humanities:

  • Good use of primary sources. Appropriate sources are chosen. They are translated and interpreted correctly.
  • Anecdotes are fine as long as the author makes it very clear how they represent general principles. 
  • Drawing connections and creating a persuasive (and beautiful) argument in favor of them. 
  • No logical fallacies.
  • Exhaustive knowledge of relevant secondary sources, both the ones who support the author's arguments and the ones who oppose them. Addressing and participating in the multi-threaded conversation among thinkers is very important in the humanities. 

Social Sciences

The social sciences try to explain human minds and human societies using scientific method, but they are limited in how they can apply scientific method because of the practical and ethical problems that arise when you try to experiment on or observe human beings, communities, and cultures under controlled conditions. 

  • As with the humanities, social science criteria for valid information sources include a good background in the work of other scholars in the subject area, and no logical fallacies. 
  • Statistics are very important for deriving measurable information from the inputs. A valid social science information source will tell you exactly what statistical instruments were used and will present not only the numbers, but also a measure of how certain those numbers are (standard deviation, plus or minus language.) 
  • Research methods are very important. A valid social science information source will have a whole section that describes methods, and will address things like how the sample was selected, how representative the sample was, and how variables were controlled for. The weaknesses and ambiguities will be addressed.
  • Avoiding bias - usually unconscious - on the part of the experimenter/observer, or in the responses of study participants - is very important. 

Sciences

The sciences pursue data about natural phenomena, with the goal of formulating theories that explain and predict those phenomena. 

  • Just like the humanities and social sciences, a science information source should include a good background in the work of other scholars in the subject area, and no logical fallacies.
  • Scientific validity requires that the claims be generalizable (externally valid), reproducible and falsifiable. 
  • Issues of metrology come up in the sciences - how accurately and precisely were they capable of measuring? How much can we trust their instruments? Were they actually measuring what they thought they were measuring?
  • Scientific research methods are extremely rigorous, even compared to social science methods. This is because scientific studies focus on phenomena that can be measured with much less ambiguity. No allowance is made for extra variables. 
  • Just like the social sciences, a science information source needs to avoid bias and use good statistical and research methods. 

Agenda and Bias = Not Valid

In an earlier section, we talked about how the intended audience and purpose of an information source can slant the information in it. The topics of agenda and bias are related to that. 

  • Ideology is a belief system shared by a group of people. Religions, political groups, and advocacy groups have ideologies. Not all ideologies are bad! And just because an author subscribes to an ideology doesn't mean that the information source necessarily has an ideological agenda.
  • Agenda is a set of goals shaped by an ideology. If an information source has an agenda, that means it's not strictly informational or educational. It's probably actually a persuasive information source or a piece of propaganda, even if it's pretending to be otherwise.
  • Bias is the tendency of an information source to selectively over-emphasize some things and de-emphasize other things in such a way that it unfairly favors a certain conclusion or point of view.

So while an author may belong to a certain ideology, that can still be OK. It's when the information source has an agenda, such that it's possible to detect bias, that's when you don't want to use the information source for research.

A few common ways that bias happens:

  • Confirmation bias is when people, including both writers and readers, have a way of not noticing or remembering things that don't fit their assumptions, and of noticing and remembering things that do fit their assumptions. (This gets even worse when Google and social media use your clicks and "likes" to decide what to show you!)
  • Confounding is when the researchers didn't make sure to rule out the effects of another variable than the one they were studying. Confounding can be either accidental, negligent, or intentional.
  • Selection bias is when you observe or do an experiment, you need to select a sample from the overall population. That sample may not be actually representative for a number of reasons. Selection bias can be accidental, negligent, or intentional.
  • Observer effect is when people (or animals) realize they're being observed, they behave differently, even if they don't mean to.
  • Publication bias is when it is really hard to get "Nothing happened," or "It didn't do what we thought it was going to do," published, so by reading the news or the scholarly literature, you get a distorted view. 

Emotional Manipulation = Not Valid

Emotional manipulation means that the information source has not stopped at trying to persuade you with facts and logic, but is actually trying to bypass your mind and get to your "heart" and your "gut." There are three main ways that this is done:

Images and imagery

It's been shown that not only do our brains process images differently than they process language; they process images faster. So a well-chosen image can get to the emotional basis for your opinions faster than logic and facts can get to the rational basis for your opinions.

Reputable news sources are supposed to try to avoid manipulating the opinions of their audiences, but in practice, they frequently do. 

Scholarly information sources are obligated to avoid manipulation with imagery. In fact, you will rarely find any images at all in scholarly sources, except photographs or illustrations of artifacts and events, sometimes portraits of persons mentioned, diagrams, charts, and graphs.

Stories

Stories work very much like images in that they appeal to our emotions, which, no matter how hard we try to be reasonable, can undermine our ability to be objective. Whenever we describe an event, we are telling a story. When the author of an information source describes something that happened in history, or in current events, or in the laboratory, the way they tell that story can end up being manipulative. 

  1. The mother parked the car, looked around the parking lot, and leaving her sick four year old son in the car, ran into the store, purchased children's cold medicine, and ran back to the car.
     
  2. Four year old Brad Wilson, suffering from a bad cold, was left alone in a car in the middle of a winter night while his mother went to the store.
     
  3. Charla Wilson's four year old son had a fever in the middle of the night, and there was no medicine in the house. There was no one to watch him. Finally Charla bundled him up, put him into his car seat, and drove to the nearest 24 hour pharmacy. She ran as fast as she could into the store, and from aisle to aisle, getting the things her sick son needed. She raced back out to the car and gave him his first dose of medicine before she even pulled out of the parking space.

As you can see, the same story, told in different ways, has a different effect on the audience. Different details are included or left out. Connections are drawn to a bigger picture, either the bigger picture of "What's a mom to do when she needs to get cold medicine for her sick kid and there's nobody to watch him?" Or the bigger picture of "How can a mom leave her sick kid alone in the car in the middle of a very cold night?" My point here isn't to decide which perspective is right or wrong. My point is that whenever you are told a story, you need to think about the parts of the story you're being asked to focus on and how you're being asked to feel about it. Also think about the parts that are being left out. 

If you find this kind of thing in an information source, it can't really be treated as credible for research.

Weasel Words and Loaded Language

Another way that information sources can manipulate your emotions is with ways of using the language to be able to imply something untrue or unacceptable, and at the same time avoid being held accountable for it. 

  • Everything in the store up to 75% off! (But most of it is only 10% off.)
  • Some authorities say that Maxatrol is the most effective cure for PMS. (But they don't say which authorities, so you can't look it up.)
  • Sugar-Sweets and Fry Fritters can be a part of a healthy diet. (No numbers are given, so you can't check them. "Can be" doesn't mean "is." And what size part? Probably a very tiny part!)

The thing about loaded language is that it's much harder to avoid than weasel words or the other forms of emotional manipulation. So when you encounter mildly loaded language, like "the policy was courageous," that's not a deal-breaker. 

But beware of information sources that sound like this (which I just got in my email) - "It's outrageous that Colgate is selling people a hygiene product that could hurt peoples' health."

A credible information source will not try to tell you how to feel about the information.

Logical Fallacies = Not Valid

Finally, logical fallacies make an information source invalid. There are literally dozens of logical fallacies. What all of them have in common is that they are like a magician's sleight of hand, except done with reasoning.

Not all logical fallacies are used intentionally. Some of them are just very common mistakes in people's thinking. That doesn't make them any more correct, though!

XKCD stick figure cartoon. Two people are talking. Panel 1: "I used to think correlation implied causation." Panel 2: That person continues, "Then I took a statistics class. Now I don't." Panel 3: The other person says, "Sounds like the class helped." And the first person says, "Well, maybe."

Some of the really common logical fallacies found in daily life:

  • False cause  
    • "I always eat pasta before an exam, and I always do well on my exams. Pasta makes me smart." 
  • Circular reasoning
    • "I know Dad's right because he says he's always right."
  • Ad hominem
    • "You're a terrible person, so your idea is wrong."
  • Slippery slope
    • "If you let the children talk back to you, soon they'll have no respect for authority and they'll be criminals." 
  • Red herring
    • "You should buy a suit at our store because shirts are buy one get one free." 
  • Appeal to authority  
    • "President Obama takes this kind of vitamin!"
  • Special pleading
    • "Well of course the dog wasn't able to bark the national anthem in front of you. You made her nervous!"
  • No True Scotsman
    • "Sports players are gentle, kindhearted people. The fact that Michael Vick abused and exploited dogs just shows that he wasn't actually a real sports player."
  • Appeal to ignorance
    • "No one knows how the Great Pyramids were built... so it must have been aliens."
  • Black or white.
    • "Liam doesn't think we should put up a veterans memorial in the park, so he must be anti-American." 
Niagara County Community College. Henrietta G. Lewis Library. 3111 Saunders Settlement Rd., Sanborn, NY 14132 | 716-614-6786