I live in the high desert mountains of the West surrounded by ranchers and cowboys. Cowboys and cowgirls are tough. Finding and evaluating sources is tough. So I'm going to teach you how to find and evaluate sources western style, like a cowboy or cowgirl. So you can be tough, too.

8.1 Why Research?

If a cowgirl wants to buy a horse, she first decides what kind she wants—there's a big difference between a hard-working quarter horse and a tall, speedy thoroughbred. Then she finds out who's selling, checks out the horse's condition, learns about its heritage, tries it out, and sees how it behaves in different situations. In other words, she does her research. We, too, spend our lives doing research—whether we realize it or not. For example, between Yelp!, Rotten Tomatoes, Consumer Reports, and Amazon reviews, it seems like we're always trying to find the best products and the best deals. When we have questions, we go to Wikipedia or ask Siri or Alexa for answers. Never in the history of the world have we had so many resources literally at our fingertips.

Quote

"Google is turning 18 years old this year. I cannot believe it was just 19 years ago that I never researched anything ever."

—Comedian Kellen Erskine, "Composed" Dry Bar Comedy on VidAngel

The trick is, how do you know you're getting the best answers to your questions? How can you tell if you're using the best sources of information or if the information you find is accurate, reliable, and up-to-date? These days you even have to ask yourself if the information you find has been planted by Russian hackers trying to influence your political opinion. The problem with a glut of resources is that it's become increasingly hard to find reliable information, which is why doing good research is becoming a necessary life skill, not just a school skill. It's especially true in the social sciences that if you know how to conduct good research, you will be more influential no matter which field you go into.

And so I give you The Cowboy and Cowgirl's Guide to Finding and Evaluating Sources with all the steps you'll need to find solid answers to your questions.

8.2 Step One: Choose Yer Horse (Select a Topic)

My kids call their grandpa "Cowboy Phil" because he wears a big custom-made hat, likes to go on cattle drives, and taught my kids how to ride a horse. He also taught them that it matters what kind of horse you choose.

 

Now stop for a second and turn on this authentic cowboy music by Brenn Hill called "Equine" about the importance of horses in a cowboy's life. Keep this on as background music as you read on. It'll really give you the cowboy/cowgirl spirit.

 



On a ranch, when the summer days get shorter and the weather turns chilly, that means it's time for the Fall Roundup. The cows pasture for the summer in the mountains eating the delicious grass and breathing the clean air. But once it gets colder, they need to be rounded up and brought back down to the ranch.

 

You can't round up cattle without a good horse, so the first step of a roundup is to gather all the horses together from the remuda (all the horses on a ranch) so everyone can collect their horses. Each cowboy or cowgirl has their string—or group—of five or six horses that each have different strengths and weaknesses. This string of horses has a lot of important jobs to do along the way—some are better at long distances, some are better at bursts of speed—but in the end, the horse you choose is the most important because it has to be your partner on your journey. As Cowboy Phil says, you choose your horse based on the task at hand. You consider your horses' strengths and weaknesses—their potential—and then choose the best one to take on the road. 

Choosing your topic for the journey of writing any kind of research paper should begin with similar consideration and care. You want to choose your research topic wisely; you don't want to invest a lot of time, energy, and resources into researching a topic for a long paper unless it has a lot of potential and is something you like enough to stick with for the long haul.

In a typical advanced writing class, you'll probably be asked to spend a significant amount of time on this topic: you'll likely create an extensive literature review looking at what others have said about your topic. You'll perhaps write a proposal based on it and/or give an oral presentation or poster presentation explaining this topic to your peers. 

So find something you love, because it won't be worth investing in unless you really love yer horse, er, topic. So what's the best way to choose a topic? Just like on a ranch, you need to do some gathering work first. Gather your string of ideas to the corral to take a good look at the possibilities. 

Background Research

One of the most overlooked and underrated parts of the research process is doing background research. Students often want to just jump right in to finding sources on a topic—grabbing the first horse they see—but the problem is that no one can know how good their topic is—especially how wide or narrow it is—until they take a good look at the bigger picture.

Getting a bird's-eye view of your topic will help you understand the context of how your topic fits into your field in general and even how important it is to the bigger world. It'll also help you discover what questions are being asked, what the hot topics are in your field, and where the most promising research is going. All this will help you create a better research question and streamline your database searches down the road. Watch this quick video on starting the research process and then we'll talk about options for doing background research.

Brainstorming

• Why did you choose your major in the first place--what topics most excited you? 
• Can you think of problems you've heard about in your field that you want to solve? 
• Can you remember something you recently heard in a class that sparked your interest?
• What's a controversy or question you want to know more about but you don't know the answer to?
• If you think back on personal problems and questions you've had in your life, how they might relate to your discipline of study?

Freewriting

Mindmapping has also been shown to jumpstart your creative juices and help your brain make new connections. There's something about thinking visually rather than just linearly that allows you to explore relationships between topics in a fresh way. In fact, we'll revisit this idea later when we talk about generating ideas for paper organization.

Mindmapping can be done on a piece of paper (the old-school preferred way) or using mindmapping apps or software like Trello, FreeMind, or XMind (new school). In the middle, write a general idea for a topic in the middle and circle it. Then draw a line to branch off that idea and write something related to it, etc. Using different colors for different branches can help you visually organize your information. You might be skeptical of this method at first, but just try it. You might be surprised what kind of ideas pop into your head when you see them visually mapped rather than simply listed in a row.

Talking

As basic as it sounds, sometimes just talking out your ideas with another person can help you make connections and discover new possibilities. Find a buddy, go to one of your professor's office hours, or even ask someone out and talk about topics you're interested in. You can either do like the freewrite and explore ideas with little judgment or you can ask the other person to give you honest feedback. Sometimes you'll be surprised what you can come up with simply by articulating your ideas to another person and hearing feedback from them.

Wikipedia and Google

Another good place to start exploring general topics is probably already one of your go-to sources: Wikipedia. In case you've been living in a cave, Wikipedia is a huge online encyclopedia containing pages that summarize millions of topics. A Wikipedia search can give you a quick sense of the history of a certain topic, who the majors players are, and what the sub-areas of research are. It can also lead you to related ideas and areas of research you hadn't though of before. Wikipedia's structure can be helpful to you as well because it breaks down larger discussion into bite-sized sub-issues and links out to other related articles. Consider how this organization can help you choose and narrow a topic. 

Because Wikipedia's not a peer-reviewed publication, you can't use it as a main source for your paper, but most information on Wikipedia includes references to the original, primary sources, so you can use their references section to help you locate reliable, peer-reviewed sources that you can use in your paper.

Doing some general Google searches will also help you see what popular sources and commercial sites address your topic and if it's been in the news lately. Trade journals by professionals in your field are also a good resource. They might have a "magazine" feel to them but deal with specific issues within an area of study.

• Which books and articles about a certain topic are the most influential?
• What's the current thinking in your field on a particular issue?
• Or what are some synonyms for specific search terms that can make database searching more fruitful (like finding out you should be searching for "emerging adult" along with "young adult")?

Note-taking

As you explore ideas about your topic, don't forget to take notes about which key words come up, how many references are listed, what sub-topics emerge, whose names you see repeatedly, and which areas of research seem especially fruitful. You'll thank yourself later in the research process when you use those notes and keywords in your database searching or if you decide you want to change your focus to a different angle or even a different topic. 

Quote

"A good background researcher sketches out main arguments, sub-topics, and specific language that’s popping up within a wider discussion during the background research stage. This language will become important later when the researcher opens up a database"

—Elise Silva, BYU Writing Programs Librarian

Here's an example of how background research helped Justin, one of my students. At first, Justin decided he wanted to study the effects of exercise on diseases or aging. But then as he did some background research, he found that there were already tons and tons of articles on that topic. So he brainstormed other ideas, talked to his peers and professors about possibilities, and even sought the help of the Economics subject librarian. 

After more exploration, Justin decided to change his focus to the influence of China on the economies of African countries. There were a lot fewer articles on that specific topic, so he knew it was a more fruitful place to put his research energy. Plus, by changing his topic early on before he spent too much time gathering sources, he saved himself a lot of time in the long run.

A Word on Changing Topics

Remember: the earlier you change your topic, the less work for you. If your horse is tired or you don't get along, that's okay. You can choose another horse from your string—that's why you brought them along. But the sooner you change the better. At a certain point, it becomes too late in the day and you have to just ride your horse to the end of the trail, tired or no. By doing good background research now, you can find that great topic fit that will take you all the way into the sunset.

 

8.3 Step Two: Saddle Yer Horse (Create a Research Question)

Research always starts with a question. It might not always be stated outright, but every time you look something up on your phone, you have a question—there's a gap in your knowledge that you want to fill. Once you have a general sense of a topic you're interested in, that's when you're ready to saddle your horse--formulate a more specific research question. Trust me, you want your horse (topic) to be saddled and ready before you get on the road or you'll be in for a bumpy ride.

Quote

“You have to find a better reason than ‘it’s an assignment’ to devote weeks to your research and for your readers to spend time reading your article. You’ll find that better reason when you can ask a question whose answer solves a problem that you can convince readers to care about. That question and problem are what will make readers think your research is worth their time.” from The Craft of Research (Booth, Colomb, and Williams, p. 35)

1. Read what's been done on a topic 
2. Figure out where the gaps of knowledge are (secondary research)
3. Fill one of those gaps with some kind of action that creates new knowledge—be it a survey, experiment, analysis, longitudinal study, product, etc. (primary research)
4. Publish the results of that action or create a prototype
5. Start the process over again

Your job right now is to do steps 1 and 2—the Secondary Research part of this process: read what's been done on a topic and figure out where the gaps of knowledge are. Then later you might even be asked to propose some primary research that could fill one of the gaps you find (although there most likely won't be time in this class to actually do the primary research, just to propose it). 

So in order to find a narrow enough topic to study and to be ready to go to the next step in the research process, you need to come up with a strong research question. You might be tempted to skip this step, but if you can take the time now to saddle your horse (by devising a narrow, specific research question), the next stretch of the research process journey—database searching—will be much, much easier. 

Watch this quick video about research questions:

Hypothesis

Although, we use the term "research question," it doesn't necessarily have to be in question format; your research question can be as simple as testing a hypothesis like this one:

"I expect that smartphone usage at a young age will increase a child's likelihood of developing anxiety." 

Even though this isn't in question format, it's an implied question—we assume that by making this statement, you'll next try to figure out whether your hypothesis is correct. So really, your research question turns into this:

"Does smartphone usage at a young age increase a child's likelihood of developing anxiety?"

A hypothesis means that you predict the answer to your research question will be "yes." You can create a more open ended approach where you adopt a posture of openness and curiosity like this: 

"I wonder if a teacher's appearance has an effect on their students."

The "I wonder" phrase is also an implied question because presumably the way you'll learn about this topic is to find answers to questions about teacher appearance and student learning. In other words, you don't have to stick to an exact formula when developing a research question, but you do need to have a specific area of inquiry in mind.

WATCO Question

If, on the other hand, you like to use formulas, one helpful way to create a research question is to formulate what my colleague Dr. Grant Boswell calls a "WATCO question." WATCO stands for What are the consequences of? Or to be more precise, What are the consequences of something (A) on something else (B)? This format can be very helpful because it forces you to narrow your topic—and I find students almost always need to narrow their topics more than expand them. As you will see as you begin to search databases for articles and books, the narrower or more specific your research question, the easier it'll be to find answers. Here are some examples of WATCO questions previous students have asked (they were assigned to come up with three possible questions).

WATCO (what are the consequences of) childhood obesity on:

1. Self confidence?
2. Adult health outcomes?
3. Society?

WATCO narcissism on:

1. Adolescent populations?
2. Adult populations?
3. Relationships?

WATCO educational funding on:

1. Student Performance?
2. Extracurricular Activities?
3. School Resources?

WATCO meritocracy on:

1. Social mobility?
2. Higher education?
3. Relationships?

WATCO bilingualism on:

1. Infants' differentiation abilities (Between the two languages)?
2. Cognizant skills as compared with their monolingual peers?
3. Infants' social skills?

WATCO a high sugar diet on:

1. Carcinogenesis?
2. Metastasis and tumor growth?
3. Cancer mortality?

Here are three different but related questions from one student:

1. WATCO dyslexia on the encoding processes?
2. WATCO early intervention with autism on ability to develop Theory of Mind?
3. WATCO speech/language therapy on patients who have suffered left-lateralized strokes?

As you can see these students are starting to narrow a large topic into smaller areas of focus. The more specific the WATCO questions, the easier it'll be for them to start the next step in the process.

Feedback

At this point, getting feedback can be very helpful. Seek feedback on your research question from your peers, your teacher, or other professors. It's likely that if they think something sounds interesting, it'll be interesting to other people as well. They can also give you ideas of ways you can narrow your topic or different avenues you can take.

Once you've gotten feedback on your questions, choose the topic that resonates the best with others and that you believe will be the most fruitful and interesting to you. Now you're ready to move on to the next stage: finding and narrowing sources (or in cowboy terms, getting the lay of the land).

More Resources

8.4 Step Three: Get the Lay of the Land (Search & Narrow)

So the first researchers who discovered cancer asked the big questions like "What is cancer?" "How does it affect the human body?" But then as they began to answer those questions, they narrowed their scope to create new knowledge. For example, cancer researchers branched their research into looking at 

• different types of cancer (e.g., What is lung cancer? What is skin cancer?) 
• or different populations (e.g., How does cancer affect people with low socioeconomic status?) 
• or different age groups (e.g., How does cancer affect children?).

By adding new factors to their research questions, they narrowed their scope and continued to create new information.

You'll follow a similar path when looking for sources about a topic: you'll start with a topic or research question, then as you search for sources, you'll probably find that you need to narrow your scope until you find that sweet spot—a manageable number of sources to take a closer look at. 

Here's a real life example: my husband is a professor here at BYU in Mechanical Engineering and Neuroscience. He studies Traumatic Brain Injury, but if he just types "Traumatic Brain Injury" into the Web of Science Database, he gets over 38,000 results: clearly too many to handle! Then if he adds another factor to narrow the scope--say, "Traumatic Brain Injury" AND "Rehabilitation"--he gets 4,800, still too many, but getting better. Finally, when he adds a third narrowing factor "AND Robotics," he gets 32 results--the Sweet Spot! That's a manageable herd. Later you'll be finding your own Sweet Spot, so stay tuned.

Finding Sources

Keywords

Library Research Guides

If you click on the Articles tab, you'll see that in the field of Economics, there are several databases that are used the most: EconLit for articles more under the general economics umbrella and Business Source Premier and ProQuest Business Collection for more business-related articles. These databases keep your search inside the relevant field and weed out the irrelevant junk you'd find just searching on Google or even Google Scholar. The librarians have already done the hunting for you, so take advantage!

Key Takeaways

• Always go through your university library website to search in databases (or even to use Google Scholar) because once you're logged in, you should have automatic access to thousands of databases. If you're not logged in, you could be locked out of many journals and other publications or be required to pay for them. If you ever have trouble accessing databases or RefWorks, contact your librarian or teacher for help.
• When you come across a perfect source that really fits your topic and you wish you had more like it, read the article's Introduction (aka Literature Review) section and check out their References page to see which articles they're citing. If they spend a lot of time talking about a certain article, you'll know it's important to their topic. You can easily search for those sources on your university library page or on Google Scholar. This is a great way to quickly find relevant sources.
• In a lot of databases, you can filter for “most cited articles” and “most cited authors” The ones that are cited the most are usually the most important sources to look at because they've had the most influence. Pay the most attention to those.

Finally, lest you think that Subject Guides are only helpful for doing research in your field, I want you to take some time to explore a subject totally unrelated to your major or research topic so you can see what's out there. In your personal life, you'll probably need to research topics that you've don't know anything about, so take a few minutes and see how much you can learn.

More Resources

If you still feel lost or would like more detailed guidance about finding sources or evaluating what you find, you can review Modules 3 and 4 here (scroll down). https://ysearch.lib.byu.edu/wrtg150/

8.5 Step Four: Round 'Em Up (Gather & Annotate)

Now that you've set your sights on the sweet spot (usually between 30-50 sources), you want to start gathering like a good cowboy. Don't just randomly grab whatever you see—you need to be selective, look at many, but only gather the best sources. The first step is to make sure you have a good rope—in other words, the right tools. 

Grab Yer Rope

"A Cowboy's most important tool is his rope." —Michele Morris in The Cowboy Life (p. 71) 

If you don't already have a RefWorks account (or other citation software), now's the time to get one. Citation Software is an easy way to save articles and other sources you find in one electronic location. The beauty of doing this is that once you've incorporated your sources into your paper, you can quickly create a References list from the software. Although you'll still need to double-check the entries it generates (always double-check!), this will still save you a lot of time. You can also use it to organize your sources and even save .pdf versions of them in your account.

There are many options for citation software, so if you have one you already know and use, you can stick with that. Some options are EndNote, Zotero, Mendeley, EasyBib, and Citation Machine. If you're a BYU student, you have free access to RefWorks, and the library has made it very easy to save items straight to your RefWorks account, so I suggest you start there. 

Here are instructions on how to set up your own RefWorks Account: https://ysearch.lib.byu.edu/wrtg150/module-3/RefWorksAccountSignup.pdf 

Here are instructions for sending documents to RefWorks: https://ysearch.lib.byu.edu/wrtg150/module-3/HowToUploadToRefWorks.pdf

If you'd like more instruction on how to use RefWorks. here's a nifty playlist of tutorial videos on YouTube: https://www.youtube.com/channel/UCzmTj_AGeY59VoNv-0SvcCg

Should you Read It?

Now comes the process of reading and evaluating your sources and deciding whether they fit your paper or not. You don't want to waste time reading articles you obviously won't need. Here are some things to quickly consider to help you decide if you should go deeper into an article. (We'll talk more about this in the next section.)

Is it credible?

Is it reliable and relevant?

How to Read Online

Reading research online is very different from reading print sources; however, more of us are reading online than ever before. Before you whip out those articles, make sure you understand how to mentally prepare yourself for the online reading experience. If you're reading an e-book or an academic article in PDF format, make sure to create a helpful reading environment for yourself by mitigating distractions, and spending a pre-determined length of time reading/annotating before jumping to the next task. Screens distract us, but deep reading, the kind you need to do when researching, does not work when you're distracted. Turn off your phone, and concentrate. 

You can also take advantage of software that allows you to Annotate (write notes directly on) articles and take notes as you go. You can do this by hand or digitally. Hypothes.is provides pretty cool software that allows you to write notes directly on online webpages and articles and also share your annotations with peers or others.

How to Read a Journal Article

Finding sources is different than understanding them—especially because academic jargon can make these texts harder to access than than what you're used to reading. Whether you're reading an academic article or an academic book, make sure you’re strategic in your approach to reading. Good researchers don't read articles cover-to-cover, so it's important to learn how to approach a source so you don't waste precious time.

First, as you're searching, look only at titles and abstracts to figure out which sources are the most relevant to your topic. Sources don't need to address all aspects of your topic to be relevant, but they should address at least one aspect that relates. When you find a source you want to take a closer look at, mark it by adding it to a RefWorks list or recording it somehow. Then don't just read straight through each article you find—that will take hours and is extremely inefficient. Instead, follow this advice from an expert:

BYU Professor Dr. E. Jeffrey Hill from the School of Family Life spoke with Wes Burr (one of the most prolific authors in the family studies field) about how to efficiently read journal articles. He offered the following suggestions (adapted and used with permission):

1. "First, read the title carefully. The author(s) likely spent many hours deciding on a title. Try to get the most understanding out of it that you can.
2. Next, read the abstract. It should contain about 50% of what you need to know from the article. Have a sheet of paper ready and draw out all of the relationships mentioned in the abstract, so you get a conceptual idea of what is going on.
3. Now read the first paragraph of the introduction. Much of what they couldn’t fit into the abstract will be in the first few paragraphs of the introduction.
4. Next read the Conclusion/Summary. This should be the take home message of the article and will give you an idea of the main points.
5. After reading the Conclusion/Summary, read each heading and first sentence of every section. You may read more if something seems particularly interesting. You may also look at figures and tables to see a succinct summary of results.
6. Next read the entire discussion section. This is where the author tells what the important findings were and the meaning of those findings.
7. Finally, if you have any time left, read the rest of the introduction to get a better idea of the background.

You almost never need to read the methods or results sections in their entirety unless you have a specific reason like if you're setting up an experiment using similar methodology (though it can be helpful to briefly look over any tables or figures). The idea is to be efficient and recognize which aspects of the article are relevant to your topic. Throughout the process, highlight key ideas and take notes. And always record which source a quote or idea comes from because you'll need that information later."

Remember that many of the conclusions any given author makes will be questioned by another source, so it’s fine if you don’t agree with what the author says—you simply need to understand the main points and how their article relates to your research and the other sources you've read.

Note-Taking

One of the most important aspects of cowboy/cowgirl life is keeping good records. It's crucial to know how many cows you have and which ones are the best. 

Just like there's an art to rounding up cattle, there's an art to sorting through sources. The goal of annotating and note-taking is for you to self-regulate—or make sure you’re understanding the piece while you’re reading it—but it’s also to create a helpful list that you can consult later as you’re writing your paper. One of the best strategies to make your life easier is to take notes as you go. You can take notes electronically on apps like Trello, on paper, on index cards, or even go old school and print out articles and highlight and write directly on them. No matter the method, you need to stay organized and be sure to keep track of which sources belong to which notes. 

While simple highlighting might be your preferred method of annotating, it doesn’t tell you much about the content of what you're reading. Focus on summarizing and noting what will be helpful as you return to the article later in the writing process. For example, if you've recorded the article's major findings as you go, it'll be much easier to sort through all your sources later when you're trying to find a detail you remember reading but don't remember where. Or if one researcher's methods are similar to another's, make note of that so you can compare and contrast them later. If you know you'll be writing a certain kind of paper like a Research Grant Proposal or a Literature Review, then keep in mind the goals of that assignment as you go. (For example, see the section on note-taking in Chapter 11: How to Write a Literature Review).

One of the best ways to stay organized (especially if you'll be writing a Literature Review on this topic in the future), is to use a color-coding system where you assign one color to each source you've gathered. Then as you read that source, summarize the most important points in your own words. As you write your summaries, record them individually and use that source's assigned color—either by writing it on a colored card/paper/post-it note, writing with different colors of ink, or if you're using electronic software, by tagging all your notes with the assigned color. This will also help you later as you try to compare or synthesize sources. It's true that you can just copy and paste good quotes as you go, but trust me that summaries will be the most valuable notes you take. Don't forget to record which page each summary/quote came from so you can easily cite it later. 

Should you accept or reject it?

Lest you think that you need to keep every source you find, watch this tutorial to see how to decide whether to accept or reject a source:

https://drive.google.com/file/d/0B2XamLhvt-6ONFRJSGYxUzNfRVE/view

Write an Annotated Bibliography

One of the most useful forms of note-taking is writing an Annotated Bibliography. Your teacher might ask you to do this in preparation for writing a longer paper. Even if it's not required, an Annotated Bibliography can be an easy way to keep track of the most important information from sources you find.

Recent research[1] has shown that college students have major difficulty summarizing articles and books (as opposed to simply paraphrasing or quoting from them). (See Chapter 9 Talking About Sources.) Summary is an important skill, and writing an Annotated Bibliography will give you fantastic practice summarizing sources. The beauty of this is that you can also use those summaries later when you write your paper and want to refer to a source. It will also help you keep track of which sources addressed which topics. In fact, if you are assigned to write a Literature Review (which you probably will be in this class), you will use summary almost exclusively.

Keep the big picture in mind as you write your annotated bibliography. Here's a good explanation of what your end goal is: to eventually synthesize your sources.

 

How to Write an Annotated Bibliography

Generally, these are the steps for writing an Annotated Bibliography:

1. Write the source alphabetized and in its full reference format (follow the appropriate documentation style such as APA, Turabian, MLA, etc.). For more help on citation formats, see Chapter 9: Talking About Sources.
2. Write a short paragraph (5-7 sentences)

a) summarizing the source,

b) evaluating its strengths and weaknesses, and

c) explaining how it's relevant to your specific topic.

For more specific information on how to best summarize sources, see Chapter 9: Talking About Sources

Additional Resources

If you’d like more guidance on writing an annotated bibliography (including examples), see Purdue Online Writing Lab’s Guide to Annotated Bibliographies (scroll down): 

https://owl.purdue.edu/owl/general_writing/common_writing_assignments/annotated_bibliographies/index.html

[1] Several internal BYU studies as well as R.M. Howard, T. Serviss, & T. K. Rodrigue. (2010). Writing from sources, writing from sentences. Writing & Pedagogy, 2.2, 177-192.

8.6 Step Five: Corral 'Em (Analyze & Evaluate)

Corralling cattle is an exhilarating practice that requires both quick thinking and strong skills on the part of the herder. For a cowboy or cowgirl to do their job right, the cattle must be found, rounded up, inspected carefully, earmarked, and then corralled—much like your sources.

Evaluate Sources

Why is this an issue?

Why do you even need to evaluate information? In the olden days (not that long ago, actually) there were texts that everyone agreed were authoritative. Maybe your parents own some old encyclopedias that they've displayed on a bookshelf. These were pretty much accepted as standard texts—you could look at them and trust the information that was printed.

Then came the internet. So much information began to be produced, it was hard to know what was credible. This has created civic debates about who believes what and what information (as people, a nation, and as communities), we should or shouldn't believe. One thing that is good about the way information is produced today is that it allows for people from all walks of life, and from all over the world to have a voice and to share it online. It's a way of democratizing access to, and the sharing of, information, points of view, and narratives that have been left out of mainstream discussions in the past.

But there is a downside: there are no longer gatekeepers of information—the editors, librarians, and experts who would fact-check information before it was produced into the encyclopedias of yester-year. This means that no matter what information you are consuming, you need to become information savvy yourself, and learn good fact-checking behaviors. This is especially true when engaging with information online, which we'll get to later in this section.

How do I evaluate scholarly material?

Scholarly material is, by its very nature, refereed before it is published. This is why academics hold it up as the "gold standard" of academic communication, and many journals engage in what is known as the Peer Review Process. Like when you peer review in class, scholars engage in a similar practice, except for instead of having a classmate review their work, academics have their work reviewed by experts in their disciplines who recommend the work be published, revised, or not published at all. This rigorous exercise is put into place to ensure strong standards in academic communication. You can find peer-reviewed journal articles through most major databases, and if you are ever confused as to whether something is peer reviewed or not, make sure to look up the journal itself online, where you can usually tell in its description whether it is the result of peer review practices.

But just because something is peer reviewed doesn't mean it's the best material for your information need. Depending on your topic, you may need to find peer reviewed material that is published within a certain time frame. It is generally accepted that the more recent the publication, the better, but this really depends. The academic conversation moves rapidly in some fields, like technology and media, and you'll need to find information published in the last 3-5 years for it to feel current. Other historical topics might allow for information to be older before it becomes dated. Seminal texts—or texts that are really important to the field—might have been published quite a while ago, but they may still be relevant to your conversation and are worthwhile to consult.

Relevance is also an important factor in determining whether scholarly information is good for your information need. This requires you to have a good sense of what you are writing about and why you are writing about it—so you can gather the best information out there. Some students fall into the "good enough" category where they just collect the first 20 sources that are good enough for them to sound somewhat educated on their topics. By so doing, they undercut the joy of the research process which is finding the best sources for their needs. But knowing what these best sources are is tricky: not every source, you see, needs to address every aspect of your exact topic to be highly relevant. Realize that every source might not talk about your whole topic--but they might talk about sub-issues within a wider topic or even related issues. 

Here is an example: say you are examining the way social media interactions affects teenage girls' behavior towards one another in person. You may find highly relevant sources that are on parts of this issue like an article on teenage social media usage, teenage social interactions, and how social media affects the brain. You might also find a few articles about sub points in your paper like cognitive development for females in teenage years. As you research, it is best to corral as many sources as you can at first, but as you cull the herd, be intentional about which ones you keep and which ones you cut loose. This will help you choose the best sources, and not just ones that are good enough.

Finally, consider other forms of scholarly communication like scholarly books. Many books published by academic presses (you can see what venue published the book in its first few pages) are peer reviewed and undergo a very rigorous editing process. To evaluate books, I'd suggest looking at who published it (a university press will carry the most clout in the scholarly communication world) and who wrote it, or contributed to it. Finally, the publication date will matter to those reviewing your bibliography.

Less scholarly sources are trade journals (like Psychology Today) which are meant for practitioners in a field. Though they're not peer reviewed, you may still find relevant research in these publications. Make sure to double check the information against the peer-reviewed research in your field before citing these sources, but do realize that they can provide a good starting point in timely, academic conversations.

How do I evaluate information I find online?

Here's where we get to the meat of it (sorry, cows!). The difficulty with information online is there is such a variety of it. We find a spectrum from highly reputable sources like the research done by the Pew Research Center—a non-governmental, not for profit, entity which studies issues relating to the United States—to enraged rantings on blog posts about the latest immigration scandal. The .org/.com tests no longer work to distinguish between "iffy" information types as just about anyone can get a .org nowadays. With opinion forums, open-access encyclopedias (read: Wikipedia) and cloak websites which hide behind layers of misinformation disguised as legitimate research to push political agendas, even the most educated of people feel wary when approaching information online. Many people would rather throw up their hands in disgust, and decide to make personal decisions about what to believe, rather than carefully learning how to evaluate information they find online for themselves, and become thoughtful researchers.

In the past we've relied on acronyms (like CRAAP: Currency, Relevance, Authority, Accuracy, and Purpose) to help give us rules of thumb about how to evaluate information; however, the information we find online often defies such definitions and quick tests of credibility. Instead, we need to learn two important lessons when interacting with information online:

1. Define your information need. This means that you need to think very carefully about what information you are seeking and go to the right places to find that kind of information. If you don't fully understand what information will fill your need, chances are you'll be looking willy-nilly for information and your information may end up coming from sketchy places. For instance, you wouldn't go to the same place to find information about a health concern you had as you would to find information about a TV you were thinking of buying. The same is the case for finding information you'll use in a paper. What is your paper about? Where would experts on this issue (either scholars, or others) be publishing? Defining your information need helps you decide where to search in the first place for good information—rather than sorting through bad information later.

For example, if you are writing about the psychology of self-driving vehicles and are trying to find a book on the subject, be warned: those may not have been written yet. Articles, which have a quicker publication rate, might be where to look for such information.

2. Act like a fact-checker. This means that you should read information online very differently than you do traditional print material. Generally folks who read a book or a print newspaper read pretty traditionally: from top to bottom/left to right. When those same people approach a web page to decide if it is credible or not, they often do the same thing: read vertically—up and down. But professional fact checkers do something very different: they read laterally. This means that instead of focusing on the content of the website or publication they are verifying—they focus on the verification. They open tabs, they double check claims—they google the folks who are behind the information and where it came from. This horizontal reading means they jump off the source to check it, rather than staying on the source and trusting what it's telling them.

Fact checkers also tend to look further down on Google results lists than students would—they realize that the first few results in a Google search can be easily manipulated, so that's why they look at results further down and look at multiple source materials about a particular issue or source before believing its claims.

Many recent studies show that students really struggle to act like fact checkers. In fact, the Stanford History Education Group recently found that when students were evaluating a politically polarizing Tweet, about half of them did not click on the link provided in the tweet to corroborate the information found therein (p. 23). This is an issue because it shows that students get caught in superficial information evaluation acts: they might notice the hyperlink is there and think that the fact that it is there alone gives the source credibility without actually clicking on it and checking it out.

Self-Evaluation: Check Yourself

Perhaps the hardest thing we need to check is ourselves and our own biases. Confirmation bias plays into source evaluation behaviors for all of us. We all have confirmation bias, which means we are drawn to and tend to support/believe sources that reinforce our preexisting thoughts. Why is this an issue while evaluating information online? Well, it's an issue because we may struggle to seek out and fairly consider sources who support points of view that are not our own.

One way to deal with this is to proactively consume online material (especially news sources) that span the ideological spectrum. For example, check out this chart below that was created by a fact-checking organization. It shows the levels of bias of most news outlets today—and its results might surprise you.

As you can see, many news sources lean one way or the other—but a whole lot are neutral and might be considered "mainstream." Such mainstream sources employ vigorous fact-checking and fact-based news reporting and are generally trustworthy.

No matter where your favorite news outlet falls on this chart, one of the best ways to avoid bias is to to double check information across several reports. When you get your information from only one source, you risk getting stuck in an echo chamber where what you hear only amplifies what you already believed. When you read more than one source, you are more likely to hear different points of view and get a more balanced view of a topic or situation. Similarly, it helps to realize that genre differences (like opinion vs. news feature) could change how you interact with the information you encounter. In sum, keep your fact-checking guard up when you read online—as Mad-Eye Moody says, "Constant vigilance, Harry!"

A second way you might deal with the issue of confirmation bias is by practicing intellectual humility. That is to say, being willing to reevaluate your beliefs, assumptions, and biases in the face of compelling, reliable, evidence. Practicing intellectual humility is not distrusting yourself or your gut, but instead, is realizing that you are a budding thinker and scholar, and you have a lot to learn. Intellectual humility is approaching research as an exercise in learning, exploring, and growing. It is an exercise in curiosity.

Revisiting one major principle: How did this information come to be?

Conclusion