Discussion Questions

2.01 Reviewing the Literature

What information in this topic best illustrates why reviewing published literature is an important part of the research-planning process? How do you plan to use this information in your review of the literature?

2.02 The Structure of Research Articles

Consider this draft title for a research report: “Improving Mathematics Achievement in Public Education.” What does the title convey? What has it left out? What feedback would you give the author?

2.03 Locating Literature in Databases

1. When you next search for journal articles, what advice from this topic will you use to approach your search?
2. If you have searched for journal articles in the past, briefly describe your experience(s). Was it easy to do? What suggestions do you have for others who will be doing it for the first time?

2.04 Connecting the Literature to Your Study

Consider ways to take notes from articles in your literature review that will help you accomplish the two primary tasks identified in this topic for connecting to your study. What information about each article would you need in your notes to be able to combine findings and summarize them by topic or theme? What details might you want to have on hand about the design of each study to compare or critique the research design?

2.05 Keeping Notes on Literature

1. What system do you use to capture your notes on references?
2. What system do you use to make citation within the document easier?
3. Do you have any tips on what has worked well in your system?

2.06 Reading and Assessing Quantitative Literature

Discuss some checks you can use on an article to determine how well the study is designed, even if you are unfamiliar with the exact statistical tests or methods used in an article. What are some strategies you might use for improving at evaluating quantitative articles?

2.07 Reading and Assessing Qualitative Literature

1. What do qualitative studies contribute to a literature review that quantitative articles cannot?
2. Why can we not typically say that qualitative studies are representative? How does that affect how we talk about or use the findings or theories qualitative researchers develop?

2.08 Using Literature to Explore Explanatory Theories

1. What is the benefit of identifying underlying theory in quantitative studies that do not directly discuss it?
2. What are the benefits of a grounded theory approach that waits to review the literature until after data collection and analysis is complete?
3. How do theories of human behavior influence our day-to-day decisions? What is an example?
4. Of the major theories that have been applied to research in the social sciences and allied fields, which one(s) appear most frequently in relation to your topic? Why might that be?

2.09 Synthesis versus Summaries

1. In your reading of the literature on your topic so far, what examples have you found of synthesizing results across multiple studies? What did you learn about your topic from them?
2. Have you noticed any strategies in the literature that writers use to paraphrase, synthesize, and summarize literature? Provide an example sentence or two and discuss how it is effective in conveying a useful synthesis or summary.

2.10 Why Academics Rely on Citation

How is citation an integral part of the formation of scientific knowledge? Why does citation matter for academic honesty or integrity? How does citation establish the authority of an author?

3.01 Concepts and Variables

1. Suppose you read a research report claiming that low-socioeconomic-status (SES) children have lower self-concepts than high-SES children do. In the report, the only definition of self-concept is "feeling good about oneself." How much credence would you give the results, in light of the definition? What additional information, if any, would you want about the definition if you were planning to replicate the study?
2. In a research report, job satisfaction is defined as "the number of times each participant said 'yes' to a set of questions on how they feel about being at work, such as 'Do you look forward to going to work most mornings?'" Is this definition completely operational? If not, what is missing from the definition?
3. Write a highly operational definition of "success in college."
4. Write a highly operational definition of "motivation to succeed on the job."
5. Review your definitions from 4 and 5. How did you create operational definitions that are multidimensional? What other factors could be important influences?

3.02 Variables in Quantitative Studies

1. Suppose you want to measure income on a self-report questionnaire that asks each participant to check off his or her income category. Name the categories you would use. Are they exhaustive and mutually exclusive? Explain.
2. Name a categorical variable that might be relevant for your study and name its categories. Are the categories mutually exclusive and exhaustive? Is it nominal or ordinal? Explain.

3.03 Variables in Experimental Studies

1. In an experiment involving which books children select at the library, name a variable that would be easy for a researcher to physically manipulate in an experiment. Then, name a variable that might be affected by the manipulation.
2. Name a variable that you would be unwilling to physically manipulate due to ethical or legal concerns.

3.04 Confounding Variables in Experimental Research

1. If you were a participant in an experiment, would you try to guess what the researcher's hypothesis was? If yes, do you think that your guess(es) would influence your behavior in the experiment? Explain.

3.05 Constructing Composite Measures

1. Have you ever given a socially desirable response that was not true of you when being interviewed or answering a questionnaire? Would you? If yes, briefly describe why. Is there anything the interviewer or questionnaire writer could have done to increase the odds that you would have been more forthright?
2. Write three statements that could be used with a Likert-type scale to measure attitudes toward your research methods class. Each should be on a different aspect of the class, such as the textbook, the instructor, and so on. Two should be positive statements and one should be negative. Explain how they would be scored.

3.06 Using Performance Measures

1. Did you ever take an achievement test on which you did not perform to your expectation? If yes, briefly describe why. Was the test maker or test administrator responsible for your less-than-optimum performance? Were you responsible? Other factors? Explain.
2. Name a specific type of achievement that can be most accurately measured using a checklist or rating scale. State why a scale or checklist would be better than a multiple-choice test.

3.07 Using Norm and Criterion Tests

1. Assume you are a parent and your child's second-grade teacher offers to provide you with either scores on a "norm-referenced test" or a "criterion-referenced test" of your child's basic math ability. Which would you choose? Why?
2. For research on the effectiveness of a technique for teaching swimming skills, which type of test would you choose? Explain your reasoning.

3.08 Falsification and the Null Hypotheses

1. The topic showed three ways to express a researcher's null hypothesis. Try writing three versions of the null hypothesis for a study of the effects of pet ownership that found people with dogs live 3 years longer on average than people with other types of pets.
2. Assuming the difference in the pet example was shown to be statistically significant, how would you describe that in terms of the null hypothesis? How would you describe it if it was found not to be statistically significant? Where are you unsure of the correct language to use to discuss the hypotheses? Bring your questions to class.

3.09 Formulating Quantitative Questions

1. Discuss the research hypotheses and statements given as examples in this topic for smartphone addiction and academic performance. Look up the articles. What additional research questions could you formulate based on these studies?
2. Restate this hypothesis as a research purpose: "It is hypothesized that there is a difference in job satisfaction between those who receive regular feedback on their job performance and those who receive irregular feedback." Is this hypothesis "directional" or "nondirectional"? Explain.
3. Restate the following hypothesis as a research question: "It is hypothesized that those who exercise regularly and those who do not exercise regularly differ in other behaviors that affect health."

4.01 Getting Started with Quantitative Research Design

Consider the research topic of teaching effectiveness used in the topic above. Consider some of the different choices that could be made based on the suggestions made in this topic. Notice the variation within the responses that you and your fellow students generate. What are some of the strengths and weaknesses in the research questions that were developed?

4.02 Types of Nonexperimental Quantitative Research

1. Could you investigate the causes of child abuse using a nonexperimental method? Which method would you use? What would be the benefits and downsides to this research approach?
2. What if you wanted to understand the effects of child abuse? What type of study would you conduct then? How would it differ from studying the causes of child abuse? What are the benefits and downsides of this study approach?
3. Suppose you read a causal-comparative study indicating that those who take vitamins A and E tend to be less overweight than the general population. What possible dangers are there in the interpretation that the vitamins cause individuals to maintain a healthy weight?

4.03 Cross-sectional and Longitudinal Designs

1. For some longitudinal studies, researchers follow people with very similar characteristics. In others, they follow a diverse group of people over the same time period. What are the upsides and downsides of each approach? If you were to design a longitudinal study, what would you study, and which strategy would you follow? Explain your choice.

4.04 Survey Research

1. Consider the various factors associated with collecting accurate and complete data. What might be most likely to affect your project’s data collection? What can you do at the planning stage that may improve your survey results?
2. Thinking about your research topic, what aspects or specific questions might be best suited to a survey method? Which are not? Explain.

4.05 Writing Survey Questions

Think of a survey you were asked to complete. (Such as a consumer satisfaction or political survey you got by email or by phone or saw posted on social media.) Did you cooperate and respond? Why or why not?

4.06 Using Existing Data

Describe a scenario when a researcher would be better off using an existing data set than collecting new data. Can you think of instances where an existing data set is the only option for a researcher?

4.07 Experimental Research

1. Suppose you read that an outbreak of intestinal disorders occurred in a town, and the source was traced to contaminated chicken served in a popular restaurant. Is it likely the research that identified the source was “experimental” or “nonexperimental”? Why?
2. Suppose you wanted to know whether having parents read to preschool children has a positive effect on the children’s subsequent reading achievement. Do you think it would be better to conduct an “experimental study” or a “nonexperimental study”? Why?

4.08 Causal Claims: Experimental Study Design

Briefly describe an experimental problem for which a researcher probably would not be able to assign participants at random to conditions.

4.09 Quasi-Experimental Designs

Suppose a researcher observed two classes for baseline data on calling-out behavior (i.e., calling out in class when it is inappropriate). The researcher observed much variation in the amount of calling out in Class A, and little variation in Class B. Which class would benefit from a longer baseline? Why?

5.01 Measures of Quality: Validity & Reliability

Suppose you were offered a choice between two midterm examinations for your research methods class. The first one contains eight short essay questions, and the second contains 38 multiple-choice questions. Both are on appropriate content. Which would you prefer to take? Why? In your opinion, which one is likely to be more reliable?

5.02 Judgmental Validity

1. Considering the limited of judgmental validity, what research areas might be more difficult for experts to determine face validity?
2. When might content validity be hard to judge?

5.03 Empirical Validity

1. Suppose a researcher validated a new multiple-choice reading test by correlating the test scores with teachers’ ratings of students’ reading abilities (i.e., the scores on the test were correlated with the ratings made by teachers). What is your opinion on using teachers’ ratings for this purpose? Could teachers’ ratings themselves be less than perfectly valid? Explain.
2. Suppose a researcher wanted to validate a new measure of self-esteem. Name a criterion that might be used in a criterion-related validity study. Be prepared to justify its use.
3. The validity of an achievement test might be established by using either content validity (see Topic 5.02) or concurrent validity (e.g., correlating the scores with teachers’ judgments of students’ achievements). In your opinion, which approach to validity is more useful for achievement tests? Should both be used?

5.04 Construct Validity

1. In your opinion, what are some of the indicators of (or behaviors associated with) the construct called industriousness?
2. To determine the construct validity of an industriousness scale, a researcher hypothesized that scores earned on it should be correlated with the number of promotions employees received on the job. Do you think this is a good hypothesis for a construct validity study? Explain.

5.05 Evaluating Reliability

1. In your opinion, which of the following variables mentioned in this topic would probably be easier to measure reliably: (1) height of customers based on observations made from a distance or (2) number of seconds from the time each customer enters a store until a salesperson greets him or her, also based on observations from a distance? Explain your choice.
2. For which of the following would test–retest reliability (with the measures administered two weeks apart) probably be more appropriate: (1) a questionnaire on investors’ opinions on the future performance of the stock market or (2) a questionnaire on prejudice against minority groups? Explain your choice.

5.06 Visualizing Reliability and Validity

If you’re learning the game of darts, why would you rather start by missing consistently just to the right of the target then having darts miss in what looks like a random pattern?

5.07 Principles of Reliability and Validity

1. Have you ever taken an achievement test in school that seemed flawed in its validity? If so, describe the test and state why you believe it was seriously flawed. In your discussion, mention the purpose for testing.
2. Consider the example of the employer who was trying to identify the best employees by using a measure of punctuality because it was reliable, even though it was not especially valid. Name one or two other traits that might be examined in evaluating employees. Comment on whether each of your suggestions can be measured reliably Also, comment on whether you think each is more valid than punctuality for determining which employees should receive bonuses.
3. Name a trait you think is elusive and thus may be difficult to measure with great validity. Discuss different ways the trait could be measured.

5.08 Threats to Internal Validity

Suppose a researcher gave a series of wellness workshops over a six-month period and then determined that five of the participants had quit smoking during the six-month period. The researcher’s interpretation was that the workshops caused the decrease in smoking. Is this interpretation flawed? Discuss your response.

5.09 Threats to External Validity

Work together to describe a hypothetical experiment on the effects of physical exercise on mood that has high internal validity but low external validity.

6.01 Populations and Samples

1. A census seeks to gather information about every individual in a population. Governments routinely conduct census counts of their citizens. How accurate can a census involving millions of people be? What types of people are difficult to find and most likely to be left out? What changes in society may affect the accuracy of the census in your country? In other countries?
2. Consider the example in the topic about studying teaching effectiveness. How would this study differ if a different unit of analysis were selected? What might change about what the researcher would be able to find, compare, or generalize to?
3. What types of research or questions would require a unit of analysis larger than individual people? When might organizations be the target population?

6.02 Simple Random and Systematic Sampling

1. Suppose a friend was planning to use simple random sampling in a research project intended to produce results that can be generalized to all students on a campus. Would you recommend drawing names from a hat or using a table of random numbers? Why?
2. Suppose a friend predicts that candidate Smith will win a local election, based on the opinions your friend has heard expressed by friends and neighbors. What would you say to help your friend understand that this method of sampling is unsound?

6.03 Stratified Random Sampling

1. Think of an issue on which you might want to conduct a survey using the students at your college or university as the population. Name the issue and two variables you think would be relevant for stratification purposes when drawing a stratified random sample. Defend your position.
2. Students were given a test on a research article they read for class. (The article reported on the use of stratified random sampling.) One test question asked, “What type of sampling was used in the research?” Students who answered using only the term “random sampling” lost a point for the question. The instructor’s comment was that the answer was ambiguous. What is ambiguous about the answer? Do you think the instructor was right in taking off the point? Why? Why not?

6.04 Cluster Sampling

To study a sample of the population of all nurses employed by 26 hospitals in a large city, a researcher drew two hospitals (clusters) at random, both of which happened, by chance, to be large public hospitals, each with hundreds of nurses, yielding a large sample. Are you impressed with the sample? Why? Why not?

6.05 Challenges in Probability Sampling

Researchers face a lot of challenges to draw a probability sample. You are probably not planning to use a probability sample, but imagine what you might do to create one for your study. Working from your target population, what techniques might you use to create a sample frame and draw randomly from it?

6.06 Nonprobability Sampling in Quantitative Research

If you could not create a probability sample because it was not possible to give all members of the population an equal nonzero chance of random selection into the study, which non-probabilistic method would you select? Why? What do you think would be the downsides of this approach? How could you offset these possible weaknesses?

6.07 Error and Bias in Samples

Are you convinced by the topic that researchers should go to great lengths to avoid bias in their samples? Why? Why not?

6.08 Modern Sampling Techniques

1. Do you ever answer calls to your cellphone from numbers you don’t recognize? Why or why not?
2. Suppose a researcher is committed to conducting a survey using a list of cellphone numbers. Brainstorm ways of increasing the rate of reaching people.

6.09 Sampling and Internet Surveys

What are some examples of groups that could yield quality samples for an Internet survey? What makes them suited to this approach? What are some of the possible problems with surveying on the Internet, and how could these issues be addressed?

6.10 Sampling and Demographics

1. Consider a survey on campus services for freshmen. Name three demographic variables that might be especially relevant for this study.
2. Are there any specific demographic characteristics that should be collected and reported in every study? What might those be? Why do they seem important?

6.11 Sample Composition and Bias

1. A magazine editor conducted a poll by printing a questionnaire in an issue of the magazine for readers to fill out and mail back. Several thousand readers returned completed questionnaires. Suppose a friend reads the results of this poll and is convinced that it reflects the views of all adults in the United States. What would you say to convince him that he might be wrong?
2. Consider the statement “The larger the sample, the better.” Explain why this statement might be misleading.

6.12 Sample Size in Quantitative Studies

Some research articles based on small samples in academic journals are characterized by their authors as pilot studies. What justifies the publication of pilot studies? Why might they be rejected?

7.01 Descriptive Statistics

Statistical concepts can be challenging. Discuss with one another the difference between descriptive and inferential statistics. Formulate applications or examples that illustrate both types of statistics and when you might use them in your research.

7.02 Mean and Standard Deviation

1. When is the mean a good measure of the central value and when is it not? Why is it questionable to use a mean for ordinal variables? When can it be helpful for ordinal variables and in what way?
2. Suppose someone gave a report and said, “The average is 25.88.” For what additional information should you ask? Why?

7.03 Median and Range

Under what circumstances is the median preferable to the mean? How can you use each of these measures of central tendency to evaluate your data?

7.04 Mode and Frequency

Try to locate a report in the popular press in which frequencies are reported. Were the data visually represented? How did they represent the data? Bring a copy to class (or share it online with the class). Compare and contrast the various ways that data frequencies are presented.

7.05 Describing Bivariate Data

Locate a report that contains cross tabulation from a research-reporting agency such as Pew Research, the census, or another data organization. Bring a copy to class and be prepared to discuss the presentation of the data.

7.06 Inferential Statistics

We often want to know what causes something, whether it is a problem like climate change or a success such as a great athletic performance. What is an example of something we might say with some certainty that we know the cause? What kinds of evidence are required?

7.07 Statistical Tests for Comparing Groups

1. Very briefly describe a hypothetical study in which it would be appropriate to conduct a one-way ANOVA, but it would not be appropriate to conduct a t-test.
2. If there are means for four groups (named A, B, C, and D), there are how many individual pairs of means to be compared with a multiple comparisons test?
3. Of the three factors that lead to a low probability when t-tests are conducted, which one is most directly under the control of a researcher?

7.08 Regression Basics

1. Consider the implications for using a regression approach. What variables might have a linear relationship? Why is the relationship linear?
2. Name two variables between which you would expect to get a strong, positive value of Pearson’s r.

7.09 Normal Distribution

1. Name a population and a variable you think might have a positive or a negative skew. Indicate which one and explain your reasons for believing that it will be skewed.
2. Locate a journal article in which the researcher reports a mean and standard deviation. Does the researcher indicate whether the underlying distribution being described is normal in shape? Do you think the 68% rule strictly applies? Discuss your reasons.

7.10 Statistical Significance

Describe the relationship between alpha, p-value, confidence level, and confidence interval.

7.11 Statistics for Effect Size

1. As noted in this topic, a small value of d might be associated with an important result. Name a specific problem that is currently confounding researchers and for which even a small value of d might indicate a result of great practical importance.
2. What is an example of when a large value of d might be associated with a difference that is unimportant?
3. In your own words, briefly explain why it is desirable to compute d when comparing the results of two experiments that use different measurement scales.

7.12 Practical Significance of Results

1. In addition to the examples in this topic, name a hypothetical result that you would favor implementing, even if it were costly, because it might make a crucial difference.
2. Consider the last research report you read that had statistically significant results. Did the researcher who wrote the article discuss the practical significance of the results? Explain.

8.01 Decisions in Qualitative Research

1. How could you turn a quantitative project that surveyed night shift nurses about their job satisfaction, family life, and health into a qualitative research project? What is likely to be similar, and what would be different about the results of the research?

8.02 Generating Qualitative Questions

Identify a “small world” you know well already. How is the group defined? What is something interesting about the subgroup that outsiders would not know? Explain what it is, what you think it means, and how you know it.

8.03 Collecting Qualitative Data

What is a research scenario for which covert observation would be critical to obtaining good data? What is a scenario in which it might be ethical to deceive a gatekeeper to gain initial access to a site for observation (assuming your purpose will be revealed in debriefing)? Would you feel comfortable engaging in deception to conduct such a study?

8.04 Mixing Collection and Analysis: Grounded Theory

1. How will you use the grounded theory method for your qualitative analysis? Discuss the principles. Do you feel that any of the principles are ones you will not use? Why? How do you think each principle affects the analysis?
2. How can you keep your analytical themes close to the data rather than imposing ideas upon the data? Brainstorm some strategies. What do you think about the controversy over the timing of the literature review? Will you review the literature before you analyze your data? What are the benefits or downsides to doing so?  

8.05 Qualitative Coding Processes and Techniques

1. Discuss the approaches to coding and the various rounds of coding. What questions do you have about each stage of coding?
2. Discuss the different programs you can use to accomplish qualitative coding. Which will you try? Why did you select this option? Consider maintaining a discussion with others involved in qualitative coding to learn the pluses and minuses of each technical approach.

8.06 Assessing Quality in Qualitative Research 

1. Considering the resources available to you, what is the most practical option for ensuring the trustworthiness of your qualitative data? What procedure(s) would best establish the consistency of your data and analysis?

8.07 Recruiting Participants for Qualitative Studies

1. Let’s say you want to conduct a qualitative research study to determine the study habits of high school honors students. Consider how you might use the sampling principles presented here. What types of things might you consider in formulating your sampling strategy?
2. Name a population (other than those mentioned in the topic) for which snowball sampling might be better than other types of sampling. Explain the reason(s) for your answer.

8.08 Sample Size and Diversity in Qualitative Research

1. The sampling methods of qualitative studies have been subject to many criticisms. What questions or concerns do you have with qualitative sampling methods? What strengths are related to qualitative approaches to sampling?

8.09 Designing Interviews

1. What is a topic or group for which a structured interview protocol is unlikely to be as effective as an unstructured approach? Why?

8.10 Conducting Interviews

1. Thinking about what it may be like to conduct interviews with total strangers, what are the biggest challenges you will face? What personality characteristics (e.g., talkative, shy, etc.) do you have that may interfere with good interviewing technique? What might you do to minimize their impact?

8.11 Planning Focus Group Facilitation

1. Which seems easier: to analyze and interpret data from individual interviews or from focus group discussions? What makes them easier or harder to analyze? What are the strengths and weaknesses of each?
2. What is a research topic that seems well suited to using focus groups? Why?

8.12 Observation and Ethnography

1. Assuming you had the time and opportunity, what is the “small world” you would most like study through extended field research? What about the people or experience most interests you? Why might it be better suited to field work than other research methods?

8.13 Designing Case Study Research

1. Do you think a method with only one or two cases can produce results on which generalizations or a theory can be based? Why or why not? What could a researcher do to improve the quality and integrity of a case study research project?
2. Why do you think fields such as medicine and business rely more than others on case studies?

8.14 Mixed Methods Designs

1. What would be a research question that is suited to the use of a mixed methods design, keeping in mind the different possible types of designs presented? What would be the contribution of each method?