8

Finding & Evaluating Sources

&

Chapter header

Learning Outcomes

In this chapter, you will learn how to

  • conduct background research on a topic
  • select and narrow a research topic and create a research question
  • utilize library and electronic resources to locate relevant information
  • evaluate sources' reliability and usefulness
  • take notes and create an annotated bibliography

Note: this chapter contains a Library Research Assignment as part of the practice for this section, so budget extra time to choose and narrow your topic.

The_Cow_Boy_1888.jpg
Cowboys are tough, just like research. (Public Domain)

I live in the high desert mountains of the West surrounded by ranchers and cowboys. Cowboys and cowgirls are tough. Finding and evaluating sources is tough. So I'm going to teach you how to find and evaluate sources western style, like a cowboy or cowgirl. So you can be tough, too.

 

8.1 Why Research?

If a cowgirl wants to buy a horse, she first decides what kind she wants—there's a big difference between a hard-working quarter horse and a tall, speedy thoroughbred. Then she finds out who's selling, checks out the horse's condition, learns about its heritage, tries it out, and sees how it behaves in different situations. In other words, she does her research. We, too, spend our lives doing research—whether we realize it or not. For example, between Yelp!, Rotten Tomatoes, Consumer Reports, and Amazon reviews, it seems like we're always trying to find the best products and the best deals. When we have questions, we go to Wikipedia or ask Siri or Alexa for answers. Never in the history of the world have we had so many resources literally at our fingertips.

Quote

"Google is turning 18 years old this year. I cannot believe it was just 19 years ago that I never researched anything ever."

—Comedian Kellen Erskine, "Composed" Dry Bar Comedy on VidAngel

The trick is, how do you know you're getting the best answers to your questions? How can you tell if you're using the best sources of information or if the information you find is accurate, reliable, and up-to-date? These days you even have to ask yourself if the information you find has been planted by Russian hackers trying to influence your political opinion. The problem with a glut of resources is that it's become increasingly hard to find reliable information, which is why doing good research is becoming a necessary life skill, not just a school skill. It's especially true in the social sciences that if you know how to conduct good research, you will be more influential no matter which field you go into.

And so I give you The Cowboy and Cowgirl's Guide to Finding and Evaluating Sources with all the steps you'll need to find solid answers to your questions.

8.2 Step One: Choose Yer Horse (Select a Topic)

Montana_cowgirl_(1909)_by_unknown.jpg
A Cowgirl wants to choose the best horse for the job, just like you should choose your best topic. (Public Domain)

My kids call their grandpa "Cowboy Phil" because he wears a big custom-made hat, likes to go on cattle drives, and taught my kids how to ride a horse. He also taught them that it matters what kind of horse you choose.

 

Quote

"You pick your horse based on the work at hand. But above all, you've got to really know your horse and make sure it knows you're the boss."

—Cowboy Phil

 
Now stop for a second and turn on this authentic cowboy music by Brenn Hill called "Equine" about the importance of horses in a cowboy's life. Keep this on as background music as you read on. It'll really give you the cowboy/cowgirl spirit.
 

Watch on YouTube
Authentic Cowboy Musician Brenn Hill sings "Equine." https://youtu.be/QPiWJzfqejY
On a ranch, when the summer days get shorter and the weather turns chilly, that means it's time for the Fall Roundup. The cows pasture for the summer in the mountains eating the delicious grass and breathing the clean air. But once it gets colder, they need to be rounded up and brought back down to the ranch.
 
You can't round up cattle without a good horse, so the first step of a roundup is to gather all the horses together from the remuda (all the horses on a ranch) so everyone can collect their horses. Each cowboy or cowgirl has their string—or group—of five or six horses that each have different strengths and weaknesses. This string of horses has a lot of important jobs to do along the way—some are better at long distances, some are better at bursts of speed—but in the end, the horse you choose is the most important because it has to be your partner on your journey. As Cowboy Phil says, you choose your horse based on the task at hand. You consider your horses' strengths and weaknesses—their potential—and then choose the best one to take on the road. 
Horses_in_Pasture.jpg
How do you choose the best horse for your journey? Photo by Vidar Nordli-Mathisen on Unsplash
Choosing your topic for the journey of writing any kind of research paper should begin with similar consideration and care. You want to choose your research topic wisely; you don't want to invest a lot of time, energy, and resources into researching a topic for a long paper unless it has a lot of potential and is something you like enough to stick with for the long haul.

Quote

"A good horse costs just as much to feed as a bad one." —Cowboy Wisdom

In a typical advanced writing class, you'll probably be asked to spend a significant amount of time on this topic: you'll likely create an extensive literature review looking at what others have said about your topic. You'll perhaps write a proposal based on it and/or give an oral presentation or poster presentation explaining this topic to your peers. 

So find something you love, because it won't be worth investing in unless you really love yer horse, er, topic. So what's the best way to choose a topic? Just like on a ranch, you need to do some gathering work first. Gather your string of ideas to the corral to take a good look at the possibilities. 

Background Research

nature-bird-wing-wildlife-beak-eagle-383325-pxhere.com.jpg
First, get a birds-eye view of your topic. (Public Domain)

One of the most overlooked and underrated parts of the research process is doing background research. Students often want to just jump right in to finding sources on a topic—grabbing the first horse they see—but the problem is that no one can know how good their topic is—especially how wide or narrow it is—until they take a good look at the bigger picture.

Getting a bird's-eye view of your topic will help you understand the context of how your topic fits into your field in general and even how important it is to the bigger world. It'll also help you discover what questions are being asked, what the hot topics are in your field, and where the most promising research is going. All this will help you create a better research question and streamline your database searches down the road. Watch this quick video on starting the research process and then we'll talk about options for doing background research.

Watch on YouTube
https://youtu.be/9vkpCvGxgqw

Brainstorming

Maybe you're like many of my students and don't even have an idea of what to research or don't really know where to start. Background research is also perfect for exploring topics and thinking about how your interests can morph into a great research question. For example, here are some questions you can ask yourself as part of the brainstorming process:
If you want to be more creative or free in your topic exploration, consider these next strategies: 

Freewriting

An easy way to generate ideas is to do a good ol' freewrite—get out a piece of paper or open a blank document on your computer and time yourself for 5 minutes.
Blue_Typewriter_Medium.jpeg
You can do a freewrite on anything from a post-it note to a napkin to a retro typewriter. Photo by Jen Theodore on Unsplash
 
Start writing about topic ideas and don't let yourself stop writing until the timer goes off. No erasing, no judgment, just keep writing. If you run out of things to say, then write, "I can't think of anything to say" until something pops into your head. The trick is to keep writing. You'll often find that ideas start to flow when you suspend judgment like this and just let whatever comes to mind flow out of your pen/keyboard. This is also a good technique to use if you get stuck in the drafting process. Studies have shown that freewriting unlocks the creativity of your brain and helps you think of connections you wouldn't normally make. In fact, some studies have shown it can actually improve your health (see, for instance, Murray, 2002 ), so  try it!

Mindmapping

Mindmapping has also been shown to jumpstart your creative juices and help your brain make new connections. There's something about thinking visually rather than just linearly that allows you to explore relationships between topics in a fresh way. In fact, we'll revisit this idea later when we talk about generating ideas for paper organization.

Mindmapping can be done on a piece of paper (the old-school preferred way) or using mindmapping apps or software like Trello, FreeMind, or XMind (new school). In the middle, write a general idea for a topic in the middle and circle it. Then draw a line to branch off that idea and write something related to it, etc. Using different colors for different branches can help you visually organize your information. You might be skeptical of this method at first, but just try it. You might be surprised what kind of ideas pop into your head when you see them visually mapped rather than simply listed in a row.

Mindmap of DST.jpg
Here's a mindmap for exploring the topic of Digital Storytelling. Photo by Robyn Jay (CC BY-SA 2.0)

Talking

Elderly_Couple_Talking_Medium.jpg
I bet this couple is talking about some awesome research topic possibilities. Photo by Cristina Gottardi on Unsplash

As basic as it sounds, sometimes just talking out your ideas with another person can help you make connections and discover new possibilities. Find a buddy, go to one of your professor's office hours, or even ask someone out and talk about topics you're interested in. You can either do like the freewrite and explore ideas with little judgment or you can ask the other person to give you honest feedback. Sometimes you'll be surprised what you can come up with simply by articulating your ideas to another person and hearing feedback from them.

 

Wikipedia and Google

Stacks_of_Books.jpg
Wikipedia has more pages than all these books put together. Photo by Ajda Berzin on Unsplash.

Another good place to start exploring general topics is probably already one of your go-to sources: Wikipedia. In case you've been living in a cave, Wikipedia is a huge online encyclopedia containing pages that summarize millions of topics. A Wikipedia search can give you a quick sense of the history of a certain topic, who the majors players are, and what the sub-areas of research are. It can also lead you to related ideas and areas of research you hadn't though of before. Wikipedia's structure can be helpful to you as well because it breaks down larger discussion into bite-sized sub-issues and links out to other related articles. Consider how this organization can help you choose and narrow a topic. 

Because Wikipedia's not a peer-reviewed publication, you can't use it as a main source for your paper, but most information on Wikipedia includes references to the original, primary sources, so you can use their references section to help you locate reliable, peer-reviewed sources that you can use in your paper.

 
Google_and_MandMs_Small.jpeg
Feel free to spell out the word "Google" in M&M's for extra inspiration. Photo by lalo Hernandez on Unsplash

Doing some general Google searches will also help you see what popular sources and commercial sites address your topic and if it's been in the news lately. Trade journals by professionals in your field are also a good resource. They might have a "magazine" feel to them but deal with specific issues within an area of study.

 

Libraries

Library.jpg
Libraries are buildings that hold books. Try one! You might like it. Photo by Shunya Koide on Unsplash
Another option is to (wait for it . . .) physically walk into the library! Did you know people publish field-specific encyclopedias and bibliographies and even subject list books—rich resources that are often overlooked (because, you know, you have to walk into the library to see them). Do it! 
 
These field-specific books can answer questions that a general internet search can't—questions like these:
It's like finding a research shortcut just by taking a walk! You can usually find these in the social sciences section of the library in their reference section. Or you can ask any librarian to direct you to these and other rich background sources that are great for exploring topics in your field. Plus, you might get in a few extra steps for the day.
 
 

Note-taking

As you explore ideas about your topic, don't forget to take notes about which key words come up, how many references are listed, what sub-topics emerge, whose names you see repeatedly, and which areas of research seem especially fruitful. You'll thank yourself later in the research process when you use those notes and keywords in your database searching or if you decide you want to change your focus to a different angle or even a different topic. 

Quote

"A good background researcher sketches out main arguments, sub-topics, and specific language that’s popping up within a wider discussion during the background research stage. This language will become important later when the researcher opens up a database"

—Elise Silva, BYU Writing Programs Librarian

Pink_Notes_Heart.jpg
Stay organized. If you want to collect your notes into a giant pink heart, go for it. You do you. Photo by Adam Kring on Unsplash

Here's an example of how background research helped Justin, one of my students. At first, Justin decided he wanted to study the effects of exercise on diseases or aging. But then as he did some background research, he found that there were already tons and tons of articles on that topic. So he brainstormed other ideas, talked to his peers and professors about possibilities, and even sought the help of the Economics subject librarian. 

After more exploration, Justin decided to change his focus to the influence of China on the economies of African countries. There were a lot fewer articles on that specific topic, so he knew it was a more fruitful place to put his research energy. Plus, by changing his topic early on before he spent too much time gathering sources, he saved himself a lot of time in the long run.

A Word on Changing Topics

Remember: the earlier you change your topic, the less work for you. If your horse is tired or you don't get along, that's okay. You can choose another horse from your string—that's why you brought them along. But the sooner you change the better. At a certain point, it becomes too late in the day and you have to just ride your horse to the end of the trail, tired or no. By doing good background research now, you can find that great topic fit that will take you all the way into the sunset.
Horse_Sunset.jpg
Do good background research now so you can ride your topic into the sunset. Photo by Helena Lopes on Unsplash
 
 

Background Research

Now it's your turn. Brainstorm some ideas and choose a general topic that interests you. (Note that for the purposes of this class, your topic should be related to your major or at least fall under the social sciences umbrella.) Do some background research: read about it on Wikipedia, search it on Google, see if it's been in the news. Pay attention to what sub-areas emerge and who the major players are. Maybe do a freewrite or create a mind map to generate more ideas. Then write two or three possibilities for a research topic you want to explore further in this class.

8.3 Step Two: Saddle Yer Horse (Create a Research Question)

Saddle_on_Horse.jpg
Like a saddle, a good research question is a helpful tool that can help you rein in your topic. Photo by Jonathan Cosens Photography - JCP on Unsplash

Research always starts with a question. It might not always be stated outright, but every time you look something up on your phone, you have a question—there's a gap in your knowledge that you want to fill. Once you have a general sense of a topic you're interested in, that's when you're ready to saddle your horse--formulate a more specific research question. Trust me, you want your horse (topic) to be saddled and ready before you get on the road or you'll be in for a bumpy ride.

Quote

“You have to find a better reason than ‘it’s an assignment’ to devote weeks to your research and for your readers to spend time reading your article. You’ll find that better reason when you can ask a question whose answer solves a problem that you can convince readers to care about. That question and problem are what will make readers think your research is worth their time.” from The Craft of Research (Booth, Colomb, and Williams, p. 35)

In academia and even in industry, there's a pattern to research:
  1. Read what's been done on a topic 
  2. Figure out where the gaps of knowledge are (secondary research)
  3. Fill one of those gaps with some kind of action that creates new knowledge—be it a survey, experiment, analysis, longitudinal study, product, etc. (primary research)
  4. Publish the results of that action or create a prototype
  5. Start the process over again

Your job right now is to do steps 1 and 2—the Secondary Research part of this process: read what's been done on a topic and figure out where the gaps of knowledge are. Then later you might even be asked to propose some primary research that could fill one of the gaps you find (although there most likely won't be time in this class to actually do the primary research, just to propose it). 

Dog_with_Question.jpg
Even this dog has a question. Photo by Camylla Battani on Unsplash

So in order to find a narrow enough topic to study and to be ready to go to the next step in the research process, you need to come up with a strong research question. You might be tempted to skip this step, but if you can take the time now to saddle your horse (by devising a narrow, specific research question), the next stretch of the research process journey—database searching—will be much, much easier. 

Watch this quick video about research questions:

Watch on YouTube

Hypothesis

Although, we use the term "research question," it doesn't necessarily have to be in question format; your research question can be as simple as testing a hypothesis like this one:

"I expect that smartphone usage at a young age will increase a child's likelihood of developing anxiety." 

Even though this isn't in question format, it's an implied question—we assume that by making this statement, you'll next try to figure out whether your hypothesis is correct. So really, your research question turns into this:

"Does smartphone usage at a young age increase a child's likelihood of developing anxiety?"

A hypothesis means that you predict the answer to your research question will be "yes." You can create a more open ended approach where you adopt a posture of openness and curiosity like this: 

"I wonder if a teacher's appearance has an effect on their students."

Child_with_Smartphone.jpg
Will this child's smartphone usage increase his anxiety? Photo by Diego Passadori on Unsplash

The "I wonder" phrase is also an implied question because presumably the way you'll learn about this topic is to find answers to questions about teacher appearance and student learning. In other words, you don't have to stick to an exact formula when developing a research question, but you do need to have a specific area of inquiry in mind.

WATCO Question

If, on the other hand, you like to use formulas, one helpful way to create a research question is to formulate what my colleague Dr. Grant Boswell calls a "WATCO question." WATCO stands for What are the consequences of? Or to be more precise, What are the consequences of something (A) on something else (B)? This format can be very helpful because it forces you to narrow your topic—and I find students almost always need to narrow their topics more than expand them. As you will see as you begin to search databases for articles and books, the narrower or more specific your research question, the easier it'll be to find answers. Here are some examples of WATCO questions previous students have asked (they were assigned to come up with three possible questions).

Formula.jpg
Do you like formulas? Then WATCO Questions are for you. Photo by ThisisEngineering RAEng on Unsplash

WATCO (what are the consequences of) childhood obesity on:

  1. Self confidence?
  2. Adult health outcomes?
  3. Society?

WATCO narcissism on:

  1. Adolescent populations?
  2. Adult populations?
  3. Relationships?

WATCO educational funding on:

  1. Student Performance?
  2. Extracurricular Activities?
  3. School Resources?

WATCO meritocracy on:

  1. Social mobility?
  2. Higher education?
  3. Relationships?

WATCO bilingualism on:

  1. Infants' differentiation abilities (Between the two languages)?
  2. Cognizant skills as compared with their monolingual peers?
  3. Infants' social skills?

WATCO a high sugar diet on:

  1. Carcinogenesis?
  2. Metastasis and tumor growth?
  3. Cancer mortality?

Here are three different but related questions from one student:

  1. WATCO dyslexia on the encoding processes?
  2. WATCO early intervention with autism on ability to develop Theory of Mind?
  3. WATCO speech/language therapy on patients who have suffered left-lateralized strokes?

As you can see these students are starting to narrow a large topic into smaller areas of focus. The more specific the WATCO questions, the easier it'll be for them to start the next step in the process.

Feedback

At this point, getting feedback can be very helpful. Seek feedback on your research question from your peers, your teacher, or other professors. It's likely that if they think something sounds interesting, it'll be interesting to other people as well. They can also give you ideas of ways you can narrow your topic or different avenues you can take.

Research Question & Feedback

Try to formulate three possible research questions based on the topic you've selected. If you want to use the WATCO format, you can choose at least one A term and three B terms or write three separate questions. Narrow your terms as much as possible. Post your own research question then comment on 3 of your peers' questions by clicking "Reply." Here's what to comment on for each question:

  1. Rate your peer's questions on a scale of 1-10 based on how much you would want to read a paper about that topic.
  2. Explain what interests you about that question.
  3. Add suggestions for improvement.

Once you've gotten feedback on your questions, choose the topic that resonates the best with others and that you believe will be the most fruitful and interesting to you. Now you're ready to move on to the next stage: finding and narrowing sources (or in cowboy terms, getting the lay of the land).

More Resources

If you'd like more help doing background research or choosing a topic, watch Modules 1 and 2 in the iframe below (you can click directly in this frame like a website). https://ysearch.lib.byu.edu/

8.4 Step Three: Get the Lay of the Land (Search & Narrow)

spring-shower-1931.jpg
In order to understand the state of your field on a topic, you need to step back and get the lay of the land. (Public Domain)
When people started exploring the West, they started with the big questions: How big is this country? Where are the mountains and rivers? What's the best path to the ocean? Then as people moved into certain areas, they narrowed their scope as their knowledge grew: Where's the most fertile soil in this valley? Where can I build my ranch to be out of the wind? Where are the best trails to take my cattle?
 
Researchers in every field also start big and get more narrow as they go. They begin by asking big questions, and as people make discoveries throughout the years, they narrow their scope to create new knowledge.
Funnel.jpg
The progression of research goes from bigger questions to narrower. (Public Domain)

So the first researchers who discovered cancer asked the big questions like "What is cancer?" "How does it affect the human body?" But then as they began to answer those questions, they narrowed their scope to create new knowledge. For example, cancer researchers branched their research into looking at 

By adding new factors to their research questions, they narrowed their scope and continued to create new information.

 
Hats_on_Wall.jpg
Too many choices, like too big a topic, can be paralyzing. Use strategies to narrow your options. Photo by Megan Markham on Unsplash

You'll follow a similar path when looking for sources about a topic: you'll start with a topic or research question, then as you search for sources, you'll probably find that you need to narrow your scope until you find that sweet spot—a manageable number of sources to take a closer look at. 

Here's a real life example: my husband is a professor here at BYU in Mechanical Engineering and Neuroscience. He studies Traumatic Brain Injury, but if he just types "Traumatic Brain Injury" into the Web of Science Database, he gets over 38,000 results: clearly too many to handle! Then if he adds another factor to narrow the scope--say, "Traumatic Brain Injury" AND "Rehabilitation"--he gets 4,800, still too many, but getting better. Finally, when he adds a third narrowing factor "AND Robotics," he gets 32 results--the Sweet Spot! That's a manageable herd. Later you'll be finding your own Sweet Spot, so stay tuned.

Sweet_Spot_of_Research.jpg
You want to narrow until you find the Sweet Spot of Research: 30-50 sources. Image by the author.

Finding Sources

So now it's your turn to get the lay of the land and start rounding up sources the way a cowboy rounds up his cattle. Where do you search? If you're a college student, you're lucky! Your university library most likely gives you free online access to specialized databases, subject guides, and hundreds of other resources, and it's all free of charge! If you're not in college, you can still check out your local library and Google Scholar for many resources. Here's where to start:

Keywords

Watch on YouTube
https://youtu.be/3Y0dceYcRH8
As you saw in the video, you'll need to use keywords to search for sources that fit your topic. Don't forget to use Boolean Operators (AND, OR, NOT) to get the best results. I hope you noted some good keywords as you did your background research—especially pay attention to the keywords listed after the Abstract of an article because these are the best terms for database searching. Remember that another good place to find synonyms for keywords is the field-specific subject encyclopedias mentioned in the last section. Sometimes if you can't find very many sources on a particular subject, it's not because no one's published on it but because you're using the wrong keywords.  

Library Research Assignment Part 1

Complete the parts of this assignment to get started navigating the library website and searching databases for your research topic.

  1. Write down your narrowed research question or topic (it should be related to your major or minor).
  2. List 3-5 possible keywords that will help you research your Research Question/Topic.
 

Library Research Guides

Speaking of resources, I want to give extra attention to probably the best resource for preliminary research: Library Research Guides. Subject Librarians are librarians who specialize in a specific field of study and have a master's degree or PhD in that subject area plus a master's degree or PhD in library and information science. They really know their stuff. These subject librarians usually work at university libraries and gather resources about specific majors or fields and find the best databases and resources for locating sources in those specific fields. In other words, they do the heavy lifting for you, so all you have to do is follow their recommendations. 
 
Here's how to do it: Look on your university's main Library website under "Research Guides." Here's an example below from BYU that is free for anyone to use. Click on the button "Research Guides" and read through the different subjects available. https://lib.byu.edu/
 

Now find your major or your field of study and click on it (you can do this right inside the iframe above). For example, if you're a Geography major, click on the Geography link, or if you're a Latin American Studies major, click on Latin American and Iberian Studies. If your major isn't listed or if your research topic is outside of your major, then find the closest subject to your topic. Once you click, you'll see tabs for finding the best types of sources like articles, books, statistics, etc. in that field. The top item in a list is the best and/or most used database in your field. There's usually even a picture and contact info for your particular Subject Librarian and a place where you can email them or set up an appointment in case you have further questions. They've thought of everything!
 
Let's take my student Justin as an example. Justin was a Social Science Teaching major, but even though his major falls under the category of Education, the topic he chose to research was the influence of China on African economies. Therefore, the Research Guide he clicked on was Economics. If you go to the Research Guide for Economics (see below), you'll find an amazing page written by the Subject Librarian with tabs listing resources for finding articles, books, statistics, and other valuable tips—resources you wouldn't have found simply typing your topic into Google or even Google Scholar. 
 

If you click on the Articles tab, you'll see that in the field of Economics, there are several databases that are used the most: EconLit for articles more under the general economics umbrella and Business Source Premier and ProQuest Business Collection for more business-related articles. These databases keep your search inside the relevant field and weed out the irrelevant junk you'd find just searching on Google or even Google Scholar. The librarians have already done the hunting for you, so take advantage!

Library Research Assignment Part 2

  1. Go to the BYU Library home page at lib.byu.edu and click on “Research Guides.” From the “Subject Guide” list, choose which subject fits your topic. List the subject and subject librarian (you might need to click on an extra tab like one that says Get Help).

  2. List the top indexes or databases for journal articles that the Subject Guide recommends you use for this topic. (List 3-5)

Key Takeaways

You'll learn more about finding sources if your class attends a live Library Session with a Subject Librarian, but here are some of the most important takeaways:
Also note that if you have questions, you can click on the Get Help tab to find the subject librarian's picture and contact information so you can request extra help. 

Library Research Assignment Part 3

  1. Do a thorough search of the Subject Guide. Explore several of the tabs. Look for resources you can use to get background information, bibliographies, statistics, and other research in your field. Note the resources you can use in future research projects. List three sources BESIDES indexes and databases that could help you in this project. Wise students use specialized background sources as well as articles and books during the research process.

  2. What is a new resource that you learned about in this search?

Finally, lest you think that Subject Guides are only helpful for doing research in your field, I want you to take some time to explore a subject totally unrelated to your major or research topic so you can see what's out there. In your personal life, you'll probably need to research topics that you've don't know anything about, so take a few minutes and see how much you can learn.

Library Research Assignment Part 4

Spend a few minutes researching a subject guide in a field very different from your major. Turn up your curiosity, open up tabs, and browse. List two things you learned in your browsing.

 
Now you're ready to get started on your search for sources. If you're lucky, your class will be meeting with a Subject Librarian. Even if not, you can try contacting a subject librarian at your library. Bring your narrowed research question, your list of keywords, and the databases you plan to explore. It's helpful to try some preliminary searches with your keywords before this meeting so you know generally how many sources are available on your topic and whether you need to narrow your scope (which is most likely). If you still have too many sources appearing, you can narrow your search by adding more factors (AND) or by leaving out some results (NOT). If you use OR, then that will expand your search, but that can be helpful if there is more than one accepted term for one of your keywords. 
 
Your goal in the end will be to find that sweet spot where you have just the right-sized herd. For a literature review where you'll probably need 15-20 sources in the end, I tell my students to gather about 30-50 sources to take a closer look at. You'll inevitably throw some out once you read them more thoroughly, so this way you can still have enough in the end. For a proposal or an infographic assignment, you might only need 5-10 sources, so gather about 20 to start with. Catch my drift? Bring any questions or problems you've had in your preliminary searches to your meeting with the Subject Librarian, and you should be ready to ride.

More Resources

If you still feel lost or would like more detailed guidance about finding sources or evaluating what you find, you can review Modules 3 and 4 here (scroll down). https://ysearch.lib.byu.edu/wrtg150/

8.5 Step Four: Round 'Em Up (Gather & Annotate)

Roundup_on_the_Sherman_Ranch,_Genesee,_Kans._Cowboy_with_lasso_readied_looks_beyond_the_herd_on_the_open_range_to_his_fe_-_NARA_-_533791.jpg
Now's the time to gather your herd and decide which sources are the best. (Public Domain)

Now that you've set your sights on the sweet spot (usually between 30-50 sources), you want to start gathering like a good cowboy. Don't just randomly grab whatever you see—you need to be selective, look at many, but only gather the best sources. The first step is to make sure you have a good rope—in other words, the right tools. 

Grab Yer Rope

"A Cowboy's most important tool is his rope." —Michele Morris in The Cowboy Life (p. 71) 

If you don't already have a RefWorks account (or other citation software), now's the time to get one. Citation Software is an easy way to save articles and other sources you find in one electronic location. The beauty of doing this is that once you've incorporated your sources into your paper, you can quickly create a References list from the software. Although you'll still need to double-check the entries it generates (always double-check!), this will still save you a lot of time. You can also use it to organize your sources and even save .pdf versions of them in your account.

There are many options for citation software, so if you have one you already know and use, you can stick with that. Some options are EndNote, Zotero, Mendeley, EasyBib, and Citation Machine. If you're a BYU student, you have free access to RefWorks, and the library has made it very easy to save items straight to your RefWorks account, so I suggest you start there. 

Here are instructions on how to set up your own RefWorks Account: https://ysearch.lib.byu.edu/wrtg150/module-3/RefWorksAccountSignup.pdf 

Here are instructions for sending documents to RefWorks: https://ysearch.lib.byu.edu/wrtg150/module-3/HowToUploadToRefWorks.pdf

If you'd like more instruction on how to use RefWorks. here's a nifty playlist of tutorial videos on YouTube: https://www.youtube.com/channel/UCzmTj_AGeY59VoNv-0SvcCg

Should you Read It?

Now comes the process of reading and evaluating your sources and deciding whether they fit your paper or not. You don't want to waste time reading articles you obviously won't need. Here are some things to quickly consider to help you decide if you should go deeper into an article. (We'll talk more about this in the next section.)

Is it credible?

Watch on YouTube
https://youtu.be/ioRGnTOqFMY

Is it reliable and relevant?

Watch on YouTube
https://youtu.be/4-I8vNdNlnI

How to Read Online

Reading_Online.jpg
Even cowgirls read online. Photo by Sincerely Media on Unsplash

Reading research online is very different from reading print sources; however, more of us are reading online than ever before. Before you whip out those articles, make sure you understand how to mentally prepare yourself for the online reading experience. If you're reading an e-book or an academic article in PDF format, make sure to create a helpful reading environment for yourself by mitigating distractions, and spending a pre-determined length of time reading/annotating before jumping to the next task. Screens distract us, but deep reading, the kind you need to do when researching, does not work when you're distracted. Turn off your phone, and concentrate. 

You can also take advantage of software that allows you to Annotate (write notes directly on) articles and take notes as you go. You can do this by hand or digitally. Hypothes.is provides pretty cool software that allows you to write notes directly on online webpages and articles and also share your annotations with peers or others.

 

How to Read a Journal Article

Finding sources is different than understanding them—especially because academic jargon can make these texts harder to access than than what you're used to reading. Whether you're reading an academic article or an academic book, make sure you’re strategic in your approach to reading. Good researchers don't read articles cover-to-cover, so it's important to learn how to approach a source so you don't waste precious time.

Reading_Frustration.jpg
Reading academic sources can be frustrating. Use smart strategies. Photo by Siora Photography on Unsplash

First, as you're searching, look only at titles and abstracts to figure out which sources are the most relevant to your topic. Sources don't need to address all aspects of your topic to be relevant, but they should address at least one aspect that relates. When you find a source you want to take a closer look at, mark it by adding it to a RefWorks list or recording it somehow. Then don't just read straight through each article you find—that will take hours and is extremely inefficient. Instead, follow this advice from an expert:

BYU Professor Dr. E. Jeffrey Hill from the School of Family Life spoke with Wes Burr (one of the most prolific authors in the family studies field) about how to efficiently read journal articles. He offered the following suggestions (adapted and used with permission):

  1. "First, read the title carefully. The author(s) likely spent many hours deciding on a title. Try to get the most understanding out of it that you can.
  2. Next, read the abstract. It should contain about 50% of what you need to know from the article. Have a sheet of paper ready and draw out all of the relationships mentioned in the abstract, so you get a conceptual idea of what is going on.
  3. Now read the first paragraph of the introduction. Much of what they couldn’t fit into the abstract will be in the first few paragraphs of the introduction.
  4. Next read the Conclusion/Summary. This should be the take home message of the article and will give you an idea of the main points.
  5. After reading the Conclusion/Summary, read each heading and first sentence of every section. You may read more if something seems particularly interesting. You may also look at figures and tables to see a succinct summary of results.
  6. Next read the entire discussion section. This is where the author tells what the important findings were and the meaning of those findings.
  7. Finally, if you have any time left, read the rest of the introduction to get a better idea of the background.

You almost never need to read the methods or results sections in their entirety unless you have a specific reason like if you're setting up an experiment using similar methodology (though it can be helpful to briefly look over any tables or figures). The idea is to be efficient and recognize which aspects of the article are relevant to your topic. Throughout the process, highlight key ideas and take notes. And always record which source a quote or idea comes from because you'll need that information later."

Remember that many of the conclusions any given author makes will be questioned by another source, so it’s fine if you don’t agree with what the author says—you simply need to understand the main points and how their article relates to your research and the other sources you've read.

Note-Taking

One of the most important aspects of cowboy/cowgirl life is keeping good records. It's crucial to know how many cows you have and which ones are the best. 

Quote

"Record-keeping is as much a part of ranch life as roping and riding." —Michele Morris, author of The Cowboy Life (1993, p. 147)

Just like there's an art to rounding up cattle, there's an art to sorting through sources. The goal of annotating and note-taking is for you to self-regulate—or make sure you’re understanding the piece while you’re reading it—but it’s also to create a helpful list that you can consult later as you’re writing your paper. One of the best strategies to make your life easier is to take notes as you go. You can take notes electronically on apps like Trello, on paper, on index cards, or even go old school and print out articles and highlight and write directly on them. No matter the method, you need to stay organized and be sure to keep track of which sources belong to which notes. 

While simple highlighting might be your preferred method of annotating, it doesn’t tell you much about the content of what you're reading. Focus on summarizing and noting what will be helpful as you return to the article later in the writing process. For example, if you've recorded the article's major findings as you go, it'll be much easier to sort through all your sources later when you're trying to find a detail you remember reading but don't remember where. Or if one researcher's methods are similar to another's, make note of that so you can compare and contrast them later. If you know you'll be writing a certain kind of paper like a Research Grant Proposal or a Literature Review, then keep in mind the goals of that assignment as you go. (For example, see the section on note-taking in Chapter 11: How to Write a Literature Review).

Color_Coding_Post_Its.jpg
Color-coding your notes will keep you organized and save you time later. Photo by You X Ventures on Unsplash

One of the best ways to stay organized (especially if you'll be writing a Literature Review on this topic in the future), is to use a color-coding system where you assign one color to each source you've gathered. Then as you read that source, summarize the most important points in your own words. As you write your summaries, record them individually and use that source's assigned color—either by writing it on a colored card/paper/post-it note, writing with different colors of ink, or if you're using electronic software, by tagging all your notes with the assigned color. This will also help you later as you try to compare or synthesize sources. It's true that you can just copy and paste good quotes as you go, but trust me that summaries will be the most valuable notes you take. Don't forget to record which page each summary/quote came from so you can easily cite it later. 

Should you accept or reject it?

Lest you think that you need to keep every source you find, watch this tutorial to see how to decide whether to accept or reject a source:

https://drive.google.com/file/d/0B2XamLhvt-6ONFRJSGYxUzNfRVE/view

 

Write an Annotated Bibliography

One of the most useful forms of note-taking is writing an Annotated Bibliography. Your teacher might ask you to do this in preparation for writing a longer paper. Even if it's not required, an Annotated Bibliography can be an easy way to keep track of the most important information from sources you find.

You already know the definition of a bibliography (a list of the sources you use in a paper, also known as a References page). To “annotate” simply means to summarize, comment on, or explain more about something (especially a text). So an annotated bibliography is a bibliography that also includes a summary and/or evaluation of each source. The annotated bibliography is so helpful that it has become a legitimate genre on its own: you can find formal annotated bibliographies on key topics published in many academic journals.
 
An annotated bibliography may seem like busy work, but it's actually a really important part of any research writing process. If done well, it helps you to gather, cull, and organize source material, get it cited correctly, and evaluate its quality. It also helps you make connections between sources and practice the art of summary and synthesis. This helps you begin to put all of these sources into conversation with one another before you actually start to compose. 

Recent research[1] has shown that college students have major difficulty summarizing articles and books (as opposed to simply paraphrasing or quoting from them). (See Chapter 9 Talking About Sources.) Summary is an important skill, and writing an Annotated Bibliography will give you fantastic practice summarizing sources. The beauty of this is that you can also use those summaries later when you write your paper and want to refer to a source. It will also help you keep track of which sources addressed which topics. In fact, if you are assigned to write a Literature Review (which you probably will be in this class), you will use summary almost exclusively.

You might also analyze and evaluate the sources in your Annotated Bibliography. These evaluations of the strengths and weaknesses of a source as well as any connections you make to other sources will help you see the bigger picture (the lay of the land) as you write. This will also come in handy as you write your paper because you can incorporate your analysis into your paper as well.
 
Keep the big picture in mind as you write your annotated bibliography. Here's a good explanation of what your end goal is: to eventually synthesize your sources.
 
Watch on YouTube
https://youtu.be/7r5bQ5ncbyQ

How to Write an Annotated Bibliography

Generally, these are the steps for writing an Annotated Bibliography:

  1. Write the source alphabetized and in its full reference format (follow the appropriate documentation style such as APA, Turabian, MLA, etc.). For more help on citation formats, see Chapter 9: Talking About Sources.
  2. Write a short paragraph (5-7 sentences)

a) summarizing the source,

b) evaluating its strengths and weaknesses, and

c) explaining how it's relevant to your specific topic.

For more specific information on how to best summarize sources, see Chapter 9: Talking About Sources

Additional Resources

If you’d like more guidance on writing an annotated bibliography (including examples), see Purdue Online Writing Lab’s Guide to Annotated Bibliographies (scroll down): 

https://owl.purdue.edu/owl/general_writing/common_writing_assignments/annotated_bibliographies/index.html

[1] Several internal BYU studies as well as R.M. Howard, T. Serviss, & T. K. Rodrigue. (2010). Writing from sources, writing from sentences. Writing & Pedagogy, 2.2, 177-192.

8.6 Step Five: Corral 'Em (Analyze & Evaluate)

Coralling_Cattle_BW_Small.jpg
Corralling your cattle is the last step in your research roundup. (Public Domain)

Corralling cattle is an exhilarating practice that requires both quick thinking and strong skills on the part of the herder. For a cowboy or cowgirl to do their job right, the cattle must be found, rounded up, inspected carefully, earmarked, and then corralled—much like your sources.

Evaluate Sources

Why is this an issue?

Why do you even need to evaluate information? In the olden days (not that long ago, actually) there were texts that everyone agreed were authoritative. Maybe your parents own some old encyclopedias that they've displayed on a bookshelf. These were pretty much accepted as standard texts—you could look at them and trust the information that was printed.

Encyclopedias.jpg
In the past, encyclopedias were simply accepted as good sources of information. We can't say as much for everything published on the internet. Photo by James L.W on Unsplash

Then came the internet. So much information began to be produced, it was hard to know what was credible. This has created civic debates about who believes what and what information (as people, a nation, and as communities), we should or shouldn't believe. One thing that is good about the way information is produced today is that it allows for people from all walks of life, and from all over the world to have a voice and to share it online. It's a way of democratizing access to, and the sharing of, information, points of view, and narratives that have been left out of mainstream discussions in the past.

But there is a downside: there are no longer gatekeepers of information—the editors, librarians, and experts who would fact-check information before it was produced into the encyclopedias of yester-year. This means that no matter what information you are consuming, you need to become information savvy yourself, and learn good fact-checking behaviors. This is especially true when engaging with information online, which we'll get to later in this section.

How do I evaluate scholarly material?

Scholarly material is, by its very nature, refereed before it is published. This is why academics hold it up as the "gold standard" of academic communication, and many journals engage in what is known as the Peer Review Process. Like when you peer review in class, scholars engage in a similar practice, except for instead of having a classmate review their work, academics have their work reviewed by experts in their disciplines who recommend the work be published, revised, or not published at all. This rigorous exercise is put into place to ensure strong standards in academic communication. You can find peer-reviewed journal articles through most major databases, and if you are ever confused as to whether something is peer reviewed or not, make sure to look up the journal itself online, where you can usually tell in its description whether it is the result of peer review practices.

But just because something is peer reviewed doesn't mean it's the best material for your information need. Depending on your topic, you may need to find peer reviewed material that is published within a certain time frame. It is generally accepted that the more recent the publication, the better, but this really depends. The academic conversation moves rapidly in some fields, like technology and media, and you'll need to find information published in the last 3-5 years for it to feel current. Other historical topics might allow for information to be older before it becomes dated. Seminal texts—or texts that are really important to the field—might have been published quite a while ago, but they may still be relevant to your conversation and are worthwhile to consult.

Relevance is also an important factor in determining whether scholarly information is good for your information need. This requires you to have a good sense of what you are writing about and why you are writing about it—so you can gather the best information out there. Some students fall into the "good enough" category where they just collect the first 20 sources that are good enough for them to sound somewhat educated on their topics. By so doing, they undercut the joy of the research process which is finding the best sources for their needs. But knowing what these best sources are is tricky: not every source, you see, needs to address every aspect of your exact topic to be highly relevant. Realize that every source might not talk about your whole topic--but they might talk about sub-issues within a wider topic or even related issues. 

Here is an example: say you are examining the way social media interactions affects teenage girls' behavior towards one another in person. You may find highly relevant sources that are on parts of this issue like an article on teenage social media usage, teenage social interactions, and how social media affects the brain. You might also find a few articles about sub points in your paper like cognitive development for females in teenage years. As you research, it is best to corral as many sources as you can at first, but as you cull the herd, be intentional about which ones you keep and which ones you cut loose. This will help you choose the best sources, and not just ones that are good enough.

cows-1029077_1920.jpg
Choose the most high-quality sources (livestock) for your herd. Photo by Pixabay from Pexels

Finally, consider other forms of scholarly communication like scholarly books. Many books published by academic presses (you can see what venue published the book in its first few pages) are peer reviewed and undergo a very rigorous editing process. To evaluate books, I'd suggest looking at who published it (a university press will carry the most clout in the scholarly communication world) and who wrote it, or contributed to it. Finally, the publication date will matter to those reviewing your bibliography.

Less scholarly sources are trade journals (like Psychology Today) which are meant for practitioners in a field. Though they're not peer reviewed, you may still find relevant research in these publications. Make sure to double check the information against the peer-reviewed research in your field before citing these sources, but do realize that they can provide a good starting point in timely, academic conversations.

How do I evaluate information I find online?

Here's where we get to the meat of it (sorry, cows!). The difficulty with information online is there is such a variety of it. We find a spectrum from highly reputable sources like the research done by the Pew Research Center—a non-governmental, not for profit, entity which studies issues relating to the United States—to enraged rantings on blog posts about the latest immigration scandal. The .org/.com tests no longer work to distinguish between "iffy" information types as just about anyone can get a .org nowadays. With opinion forums, open-access encyclopedias (read: Wikipedia) and cloak websites which hide behind layers of misinformation disguised as legitimate research to push political agendas, even the most educated of people feel wary when approaching information online. Many people would rather throw up their hands in disgust, and decide to make personal decisions about what to believe, rather than carefully learning how to evaluate information they find online for themselves, and become thoughtful researchers.

laptop-3087585_1920 (1).jpg
Don't know what to believe? Hone your source evaluation behaviors. Photo by JESHOOTS.COM on Unsplash

In the past we've relied on acronyms (like CRAAP: Currency, Relevance, Authority, Accuracy, and Purpose) to help give us rules of thumb about how to evaluate information; however, the information we find online often defies such definitions and quick tests of credibility. Instead, we need to learn two important lessons when interacting with information online:

1. Define your information need. This means that you need to think very carefully about what information you are seeking and go to the right places to find that kind of information. If you don't fully understand what information will fill your need, chances are you'll be looking willy-nilly for information and your information may end up coming from sketchy places. For instance, you wouldn't go to the same place to find information about a health concern you had as you would to find information about a TV you were thinking of buying. The same is the case for finding information you'll use in a paper. What is your paper about? Where would experts on this issue (either scholars, or others) be publishing? Defining your information need helps you decide where to search in the first place for good information—rather than sorting through bad information later.

For example, if you are writing about the psychology of self-driving vehicles and are trying to find a book on the subject, be warned: those may not have been written yet. Articles, which have a quicker publication rate, might be where to look for such information.

2. Act like a fact-checker. This means that you should read information online very differently than you do traditional print material. Generally folks who read a book or a print newspaper read pretty traditionally: from top to bottom/left to right. When those same people approach a web page to decide if it is credible or not, they often do the same thing: read vertically—up and down. But professional fact checkers do something very different: they read laterally. This means that instead of focusing on the content of the website or publication they are verifying—they focus on the verification. They open tabs, they double check claims—they google the folks who are behind the information and where it came from. This horizontal reading means they jump off the source to check it, rather than staying on the source and trusting what it's telling them.

Fact checkers also tend to look further down on Google results lists than students would—they realize that the first few results in a Google search can be easily manipulated, so that's why they look at results further down and look at multiple source materials about a particular issue or source before believing its claims.

Fact Checking Behaviors

Find a website or article you don't know much about on a social media feed. Practice expert fact checking behaviors to decide if the information is credible or not. Record a reflection on your experience.

Many recent studies show that students really struggle to act like fact checkers. In fact, the Stanford History Education Group recently found that when students were evaluating a politically polarizing Tweet, about half of them did not click on the link provided in the tweet to corroborate the information found therein (p. 23). This is an issue because it shows that students get caught in superficial information evaluation acts: they might notice the hyperlink is there and think that the fact that it is there alone gives the source credibility without actually clicking on it and checking it out.

Civic Online Reasoning Classroom Poster_blue.jpg
Stanford History Education Group's suggestions for engaging with information online. Learn more at: https://sheg.stanford.edu/civic-online-reasoning

Self-Evaluation: Check Yourself

Perhaps the hardest thing we need to check is ourselves and our own biases. Confirmation bias plays into source evaluation behaviors for all of us. We all have confirmation bias, which means we are drawn to and tend to support/believe sources that reinforce our preexisting thoughts. Why is this an issue while evaluating information online? Well, it's an issue because we may struggle to seek out and fairly consider sources who support points of view that are not our own.

One way to deal with this is to proactively consume online material (especially news sources) that span the ideological spectrum. For example, check out this chart below that was created by a fact-checking organization. It shows the levels of bias of most news outlets today—and its results might surprise you.

Media-Bias-Chart_4.0_Standard_License-min-1024x791.jpg
Levels of bias and fact checking in news sources. (Used with permission.)

As you can see, many news sources lean one way or the other—but a whole lot are neutral and might be considered "mainstream." Such mainstream sources employ vigorous fact-checking and fact-based news reporting and are generally trustworthy.

No matter where your favorite news outlet falls on this chart, one of the best ways to avoid bias is to to double check information across several reports. When you get your information from only one source, you risk getting stuck in an echo chamber where what you hear only amplifies what you already believed. When you read more than one source, you are more likely to hear different points of view and get a more balanced view of a topic or situation. Similarly, it helps to realize that genre differences (like opinion vs. news feature) could change how you interact with the information you encounter. In sum, keep your fact-checking guard up when you read online—as Mad-Eye Moody says, "Constant vigilance, Harry!"

Harry_Potter_Medium.jpg
As Mad-Eye Moody says, you need "Constant Vigilance" when reading online. Photo by Artem Maltsev on Unsplash

A second way you might deal with the issue of confirmation bias is by practicing intellectual humility. That is to say, being willing to reevaluate your beliefs, assumptions, and biases in the face of compelling, reliable, evidence. Practicing intellectual humility is not distrusting yourself or your gut, but instead, is realizing that you are a budding thinker and scholar, and you have a lot to learn. Intellectual humility is approaching research as an exercise in learning, exploring, and growing. It is an exercise in curiosity.

Revisiting one major principle: How did this information come to be?

At this point you might be a little frustrated and be thinking, "Well what can I trust? I guess I can't trust anything or anyone!" Distrust of the media is a bit of a scary phenomenon since it seems to give license for individuals to believe whatever they want rather than becoming informed on salient issues. That is why I advocate an informed approach to information consumption rather than an approach which is self-defeating. If there is one idea I want you to walk away from this section with, it is this: in order to really evaluate information, you need to figure out how that information was created; in other words, figure out how it came to be
 
Such a question as "How was this information created?" hits on many points: who is behind this information? Who wrote it? Who published it? What kind of a publication process did it go through? Was it vetted? Was it reviewed and edited? By answering these questions you'll be able to construct a picture of the process this information went through in order to come to you. You might even be able to surmise if someone had an agenda or motivation that seemed hidden at first. As with everything, you'll need to make some value judgments along the way. For example, some kinds of information will help you establish your credibility in a paper, whereas other kinds of source material will lend humor or pathos to an argument. As you engage with such sources consider your audience and what kind of information they would expect you to use for your writing. It all comes back to context when you make the decision to trust a source or to use it purposefully in your writing. 

Conclusion

Now that you've evaluated, analyzed, and started to synthesize your sources, you're ready to learn how to cite them. In other words, now that you've rounded up and corralled your herd, you're finally ready to incorporate those sources into your writing. You're living the dream!
Cowboys in Forest.jpg
This could be you living the research dream. Photo by Tobias Keller on Unsplash
Cristie Cowles Charles

Brigham Young University

Cristie Cowles Charles teaches writing and literature courses at Brigham Young University in Provo, Utah. She enjoys sparking a love for writing in her students--or at least a love for having had written (it's always worth it in the end, right?). She thinks pumpkin pie counts as a vegetable, is married to a super hot mechanical engineering and neuroscience professor (yes, they exist), and adores her five magnificent children.

Elise Silva

Elise Silva holds an MA in English from BYU and a Masters of Information Science from the University of North Texas. She is currently a PhD student at the University of Pittsburg studying composition and rhetoric. Before that, she was the academic librarian over Writing Programs at BYU.

This content is provided to you freely by Ensign College.

Access it online or download it at https://ensign.edtechbooks.org/writing/resources.