COMPREHENSIVE EXAM QUESTIONS IN RESEARCH METHODS(1)


1. PHILOSOPHY, LOGIC AND ETHICS OF SCIENCE

A) General

A1. Starbuck has suggested that "the properties shared by all organizations ought to be uninteresting and unimportant." He also notes that "although statements about averages bother very few, they ought to bother many" (Journal of Management Studies, vol. 30(6), 1993).
Do you agree with these statements? If so, why, and what are the implications of your beliefs for research methods in strategic management and organization science? If not, why not, and what are the implications of your beliefs for research methods in those areas?
TH
A2. Lay out the components of the Runkel and McGrath (1972) Research Cycle and the Martin (1982) Garbage Can Model of the research process. What are the aims of either approach? What assumptions does each approach make? What basic themes does each approach emphasize? What kinds of constraints on the process of doing research does each approach highlight? How are the two approaches alike and different in other ways? What are the likely pitfalls of viewing the research process from only one of these two viewpoints?
A3. Define (and discuss important features or issues concerning) the following constructs of Philosophy and Logic of Science. (Define any four terms).

1. Null hypothesis 6. Nomological network
2. Logical empiricism (positivism) 7. Modus tollens
3. Paradigm 8. The fallacy of affirming the consequent
4. Independent variable 9. Theory
5. Operational definition

A4. Organizational Studies may be described as either a basic or an applied discipline. From your point of view, is Organizational Studies a basic discipline or an applied discipline, or both? What should it be? When answering this question, be sure to define distinctions between these two types of disciplines. Provide evidence from the organizational studies literature that supports your position.

A5.

What rules of evidence are used to determine whether a research finding is meaningful?

A6. Considerable debate in the social and behavioral sciences has occurred concerning the role of the context of discovery versus the context of justification. Much of the debate has addressed the following sets of issues:

a. What features distinguish the role of discovery from the role of justification?

b. What is the role of discovery versus justification in the research process?

c. Compare the perspectives of a logical empiricist and a relativist. What distinctions would they make between the context of discovery and the context of justification?

d. Can a discipline be said to progress within the context of discovery?

A7. An epistemology is a set of rules for the appropriate way to generate knowledge. Outline the major components of your epistemology for Organizational Studies research. You may wish to proceed by identifying one of the established epistemologies and comparing it to yours (e.g., skepticism, positivism, empiricism, pragmatism, relativism). Describe the process by which you feel knowledge can be generated most effectively in Organizational Studies. Identify and define the key concepts of your research philosophy and discuss its strengths and weaknesses.

A8. Abraham Kaplan argues that behavioral scientists should practice "openness in inquiry"; that is, they should not pre-judge research or theory negatively or positively on the basis of its content. There are many other competing philosophical schools of thought as to how empirical research should be conducted. Much of the debate has addressed the following issues:

a. What are the criteria for judging whether research is worthwhile? Is there some research that should not even be done?

b. What is the most effective model of the research process?

c. How does a researcher determine the meaningfulness of a research finding? That is, what are the rational rules of evidence necessary for evaluating research findings.

In your doctoral studies, you have been exposed to a variety of approaches for addressing each of the above issues. Please describe, in detail, your position on these issues and provide citations where appropriate. In addition, please highlight the major competing positions on each issue.

A9. In recent years, there has been considerable debate in the social and behavioral sciences concerning the two competing epistemologies-- logical empiricism (including falsificationism) and relativism (i.e., post-positivism).

a. What characteristics distinguish these two epistemologies? Be sure to address both conceptual and methodological differences.

b. Take the perspective of a logical empiricist. How would this perspective affect the manner in which you conduct research? Be sure to discuss at least the following issues: a) what model(s) would you use to approach the research process, b) theory evaluation, and c) data collection.

c. Now take the perspective of a relativist. Address the same issues raised in part b.

A10. In recent years, there has been considerable debate in the social and behavioral sciences concerning two competing research paradigms--one based on a logical empiricism/realist epistemology and the second based on a post-empiricist epistemology. These paradigms vary with respect to their underlying aims (goals), standards for judging explanations, and ontology.

a. Describe the basic aims, standards, and ontology for each paradigm. What characteristics distinguish these two paradigms?

b. One current concern focuses on the appropriateness of a priori and post-hoc explanations for theory testing. Describe and defend your point-of-view concerning which type of explanation is appropriate for research in organizational studies.

c. A second current concern focuses on the appropriate treatment of the independent variables; that is, an experimental approach (where the independent variables are manipulated) and an observational approach (where the independent variables are observed). Describe and defend your point-of-view concerning which treatment of the independent variables is likely to produce the more important contribution to organizational studies.

A11. What rules of evidence are used by the positivist research paradigm to determine whether a research finding is meaningful? Choose another research philosophy/paradigm and compare and contrast it to the positivist rules of evidence.
A12. A major tenet of research is that research subjects should have limited, if any, knowledge of the subject of the research. The assumption is that subjects will not act naturally if they are aware of the purpose of the research. For this reason researchers often engage in various types of deception. For example, experimental and survey researchers will word instructions in such a way as to hide the research goals; and researchers who conduct participant observation will avoid telling people they are researchers. Discuss the ethical questions raised with deception. What guidelines would you give to a colleague who was conducting research that involved deception?

B) Causation

B1. Choose a model from the Organizational Behavior literature that has causal orderings between variables and has been tested in a field setting. Compare the following data analytic methods for testing hypotheses about causality between variable sets (e.g., dependent, independent, intervening, moderating, exogenous, endogenous):

a) path analysis using hierarchical regression

b) maximum likelihood estimates of structural equations models (perhaps through LISREL)

Your answer should address:

1. The parameters the procedures estimate and how they are estimated.

2. How the validity of the models are judged.

3. Appropriate inferences of causality and conditions necessary to make such inferences.

4. The statistical assumptions underlying each technique and the consequences of violating assumptions.

B2. Describe the critical-realist perspective of Cook and Campbell (1979). Compare and contrast this perspective on causality with that of Popper's falsificationism.
B3. According to Cook and Campbell (1979), causal relationships in the social sciences can be probabilistic and still meet the assumption of causality that asserts that a cause must always produce an effect. Describe the logic they invoke to justify this position.
B4. Identify James, Mulaik, & Brett's (1982) conditions for establishing causal inference. Discuss what they mean and why they are important to causal inference.
B5. What are the three conditions necessary to establish causality? Can these conditions ever be fully met? Why or why not?
B6. Assume you are interested in conducting a study that allows you to legitimately make some causal inferences concerning relationships among a set of constructs. What sorts of things can you, as the researcher, do to bolster your ability to make such causal inferences? In answering this question, be sure to pay attention to issues pertaining to both study design and data analysis.
B7. Some researchers (and editors/reviewers!) are very critical about the ease with which others use causal language in interpreting their analyses. What are the concerns behind this criticism and what are the methodological and statistical conditions that give rise to these concerns? Under what conditions can researcher make reasonable valid causal interpretations of their findings? Can a researcher ever be completely confident making such causal statements? Be sure to defend your position in answering these questions.

C) Theory & Research

C1. What are the elements of a theory? What criteria did Bacharach (1989) use to evaluate if a theory is good? IC
C2. The Academy of Management Journal, in its "Information for Contributors," states, "All articles published in the Journal must make a strong theoretical contribution." Because this is one of our top publications, we must have a clear understanding of what constitutes a strong theory. Describe the essential elements of a good theory. You can also describe practices that are not indicative of good theory if that will help you articulate what is good theory.
C3. Organizational researchers place a high premium on research that is guided by theory. This perspective is evident in journals such as AMJ and ASQ, which will often reject a manuscript that describes a sound empirical study but does not provide a theoretical contribution. Is there a role for atheoretical research in the organizational sciences? If so, what is the role of atheoretical research, and what contribution does it make to knowledge in the organizational sciences? TH
C4. Define inductive and deductive methods for generating knowledge. What are the advantages and disadvantages of each? Under what conditions is each appropriate?
C5. Daft (1985) in "Why I Recommended That Your Manuscript Be Rejected and What You Can Do About It," said that poor theoretical development is the primary reason he recommended manuscripts be rejected when reviewing for AMJ and ASQ.

Why is theory considered so important?

Give an example of a good theory and explain the elements that make it a strong theory. (Focus on theory development, not on the content of the specific theory.)

C6. You have undoubtedly heard the criticism of certain published articles in our field being "atheoretical". Spoken as a criticism, it suggests that research articles should have theory. Should every empirical article have theory? Why is theory important? What are the hallmarks of good theory?

2. RESEARCH DESIGN

A) Research Design Terms

A1. Define (and discuss important features or issues concerning) the following constructs of research design. (Define any five terms.)
1. Internal validity 6. Quasi-experimental design
2. Stratified Sampling 7. Random Assignment
3. Solomon Four-group design 8. Differential mortality
4. Factorial design 9. Demand characteristic
5. External validity 10. Regression to the mean

B) Level of Analysis

B1. I/O psychologists have traditionally categorized themselves as being more micro or macro in orientation, usually falling toward the micro end. Recent concern for what has been termed the levels of analysis or cross-level approach has added a third category - "meso."  What is the meso level?  Is this just academic hairsplitting or re-naming or is there a fundamental difference in this research strategy in contrast to micro or macro?  Provide an example from some research domain of interest to you that illustrates your conclusion of "no real difference" or "important difference."

C) Cross-Sectional vs. Longitudinal Design

C1. The use of cross-sectional research designs continues, despite many calls for replacing them with longitudinal designs. (a) What are the main strengths of cross-sectional designs? (b) What weaknesses do they have that longitudinal designs supposedly counter? (c) What are the implications of cross-sectional designs for theory development? (Consider internal and external validity.)
C2. A good number of research efforts on the micro (OB/HR) side rely on cross-sectional designs in which surveys are used to collect both IVs and DVs from the same respondent. What are the various problems associated with such a design? What changes could be made to this design to alleviate and/or eliminate these problems? What are the downsides of the new design? (add in convenience/non probability samples)
C3. What are some of the major strengths of a longitudinal design? Why don't we see more of them?
C4. Assume that you are analyzing a large sample of longitudinal survey data.

Identify two characteristics of your data that would make ordinary least squares regression (OLS) inappropriate for conducting analyses. For each of these characteristics, describe an alternative analysis that is superior to OLS.

TH

D) Qualitative vs. Quantitative Designs

D1. Compare and contrast the objectives, strengths and weaknesses of qualitative/observational methods (e.g., ethnography) versus quantitative methods (e.g., survey) in conducting empirical research on macro organizational issues. Be sure to consider issues of reliability, generalizability, validity, ethics and practicality in your answer. Describe situations where one method or the other may be most appropriate. IC
D2. In recent years a number of individuals have taken the position that qualitative research is superior to quantitative research. a. Compared with quantitative research, what are the principal strengths of qualitative research? b. Compared with quantitative research, what are the principal weaknesses of qualitative research?

E) Observation vs. Experiment

E1. Compare and contrast the objectives, strengths, and weaknesses of observational methods (e.g., ethnographic methodology) and experimental methods. Be sure to consider issues of precision, generalizability, and realism in your answer. Consider also situations where one methodology or the other might be most appropriate.

F) Laboratory vs. Field Research

F1. Of the various research approaches available to organizational researchers, laboratory experiments are held up as an ideal, because of their potential for high internal validity. On the other hand, field research, including participant observation and field experiments, have high potential for external validity. Discuss specifically the strengths and weaknesses of laboratory versus field experiments. How would positivist researchers think differently about these tradeoffs than would phenomenologists?
F2. In conducting an experiment in the laboratory or designing survey research there are certain unique opportunities and limitations of each research strategy. A) Describe a hypothesized relationship between two constructs, an independent and a dependent variable in a substantive area of your choice (e.g., Strategy, OB, HRM, Organizational Theory); provide a verbal, mathematical and graphical statement of that hypothesis. B) Briefly discuss how you would examine this hypothesized relationship in a laboratory using experimental techniques versus a field survey using correlation techniques. C) Discuss what may be considered typical strengths and weaknesses of each strategy regarding issues of precision and control in the manipulation or measurement of the variables of interest, realism of the context for the participants, and generalizability of the findings to other elements of a chosen population. D) Finally, discuss advantages, if any, of using laboratory and field survey methods in tandem.

G) Experimental Design

G1. Experimental designs oftentimes come under attack due to perceived limitations in external validity. (a) Relative to other design types, what strengths and weaknesses do they offer? (b) What experimental design structure is the strongest? Why? (c) What is the relevance of research design for results and theory development?
G2. Meta-analysts often investigate whether the quality of research design (e.g., randomized experiments vs non-randomized designs) influences average effect size. In their meta-meta analysis of psychological, educational and behavioral treatments, Lipsey and Wilson (1993) found that "In a given treatment area, poor design or low methodological quality may result in a treatment estimate quite discrepant from what a better quality design would yield, but it is almost as likely to be an underestimate as an overestimate" (p.1193). A few methodologists (e.g., Shadish) have attempted to identify factors that may explain the relative magnitude of effect sizes from randomized and nonrandomized experiments. Please review these findings, and add your tentative model of the conditions under which random experiments will produce larger, smaller, or equal effect sizes, relative to nonrandom designs. Discuss possible implications for research design in the context of "real-world" interventions.
G3. Analysis of covariance and randomized blocks designs are sometimes referred to as "noise-blocking" designs. Explain what this means and give an example of (a) a randomized blocks experiment, and (b) an analysis of covariance experiment. Keep your examples simple. Describe how such designs are "noise-blocking" when used with randomized assignment of subjects to experimental condition, and contrast that with the role such designs serve in quasi-experimental designs.
G4. Team performance in Bafko Industries varies considerably across teams and from month to month. The senior VP in charge of team development thought that a team training program could improve the performance of these teams. A consultant convinced him that his firm's training program, which lasted only one week and cost only $25,000 per team) could raise the performance of Bafko's teams. The VP, however, wanted to experimentally test the efficacy of this training program before he adopted it widely throughout the firm. So the consultant suggested the following experimental test:

There were 48 teams in Bafko that showed significant variation from month to month in their level of performance. The consultant chose the 24 worst performing teams based on last month's data and assigned them to the training group. "After all," he argued, "these were the ones who most needed it." The other teams were assigned to the control group. The training group received the training program (lasting one week) at the beginning of the next month while the control group teams did their normal routine. Performance scores were gathered at the end of the month. The results showed that the training group teams had a significantly larger rise (i.e., statistically significant) in their performance than did the control group teams (who, as a group, did not improve at all). The consultant proudly displayed these results and argued that they convincingly demonstrated the efficacy of his program.

How would you advise the VP? Specifically, what is the validity of the consultant's conclusions? If you think he might be mistaken, what specific threats to internal validity are particularly plausible?

G5. Consider the following design:

N  O  Xa  O

N  O  Xb  O

N  O  Xc  O

N  O        O where,

Xa , Xb, and Xc represent different levels of a training program that varied in its length across three different work groups. In group a the training time was 40 hours; in group b it was 30 hours; and in state c it was 20 hours. The training was designed to instill customer service values in the team members. O represents a measure of customer service attitudes measured with a 20-item Likert scale. This measure was administered one week before the training program and 3 months later. The researcher hypothesized that the training program would improve customer service attitudes, and that longer training would have stronger effects. The design is a non-equivalent groups design because people were not randomly assigned to groups, even though groups were randomly assigned to treatment condition. Each group has a sample size of 36 members. The reliability of the customer service attitude scale was .78 at the pretest and .80 at the posttest. The pretest means were as follows:

Group A: 2.6

Group B: 2.3

Group C: 2.7

Control Group: 2.5

a. Construct a regression model that will test the researcher's hypothesis. Be explicit here, describing each variable in the model, and what regression effect(s) you would expect to find if the hypothesis were correct. (Hint: everything you need to know is in Trochim's chapter 11 and what we discussed in class). (10 points)

b. Describe any serious threats to internal validity. (10 points)

c. Discuss how you might amend this design to improve its internal validity.(10 points)

G6. Often in organizational research the researcher is faced with having only one group on which to experiment, and thus cannot use a control group when conducting experiments. This occurs, for example, in taxation experiments when you cannot selectively apply new tax laws to different groups, or when you wish to test the efficacy of an incentive plan that must be applied to all members of a company. In such situations the researcher must be creative in assembling a quasi-experimental design that has high validity (and usually the challenge concerns internal validity). Consider the following scenario.

A manager of a retail store wants to test the efficacy of an incentive program in changing the selling behaviors of her retail sales clerks. Specifically, she wants to test a behavioral reinforcement program in which sales people's behavior is recorded on in-store video cameras. The tapes are analyzed each night and the experimenter tabulates the frequency of various behaviors (e.g., offering assistance to customers, restocking items, straightening up displays, suggesting accessory items to accompany the items that a customer is purchasing). The manager then rewards the clerks with a monetary bonus based on the frequency of desired behaviors. She would like to conduct a valid experiment to see what effect such an incentive program has on employee behavior, but because of concerns about employee morale, she cannot assign clerks to treatment and control groups. Instead, she has to apply the experimental treatment to everyone in the store. She learned in her research methods class (which the company requires all its managers to take) that a single group experimental design typically has very low internal validity.

Your job as a consultant is to help her design an experiment, given her constraints, that would produce interpretable (i.e., internally valid) results. Describe the details of the design you would recommend, using the notation in Trochim (2001). If the design you come up with has a specific name, use it. As part of describing your design, discuss how the data would be analyzed and what particular effect you would look for.

G7. Why are experimental research designs held in such high regard?
G8. Compare and contrast experimental, quasi-experimental, and correlational research designs. What characteristics separate these three designs? What are their advantages and disadvantages, especially in comparison to one another?
G9 What is the difference between a within-subjects experimental design and a between-subjects experimental design? When might a within-subjects design be appropriate? When might it be inappropriate?
G10. Describe the difference between a basic within subjects and between subjects experimental design. What are the advantages and disadvantages of each, especially in comparison to one another? Give an example from your area of research in which each would be appropriate.
G11. What are the major threats to the internal validity of a pretest - post test nonequivalent control group design? How can this design be improved with regard to increasing a researcher's ability to make causal inferences?
G12. A production manager is interested in testing the hypotheses that participation in quality circle programs will result in increased productivity and reduced absenteeism. A questionnaire is administered to a group of 92 employees concerning their attitudes toward such programs, and 46 with the most favorable attitudes who are willing to participate are selected for the treatment group. The remaining 46 are the control group. Measures of performance (units/month for the group) and absenteeism (lost days/month for the group) are kept over a three month period during which the treatment group has weekly quality circle meetings. The results are as follows:
Month 1 Month 2 Month 3 Month 1 Month 2 Month 3
Performance Absenteeism
Treatment 100 115 121 6 9 4
Control 110 108 113 6 8 5

The manager calculates t tests between month 1 and 3. The only significant difference is performance for the treatment group, so the manager concludes that quality circles raise productivity but do not have an effect on absenteeism. What are the primary threats to internal and external validity in this study. What are possible alternative explanations for the differences (or lack of differences) between the two groups? What additional information might help explain these results?

G13. One stream of current strategy research contains "event studies." In these studies, investigators basically correlate the reaction of a financial market to the presence and/or absence of the occurrence of some event at a firm (e.g., the resignation of a CEO). In a different approach to such a study, suppose that one investigator (Jean) "conducts" a natural field experiment on the effects of competitor mergers on a firm's environmental scanning activity. Through previous arrangement, Jean has been working with 5 medium-sized computer hardware and 5 medium- sized software firms on the west coast. She's been collecting a "paper trail" of internal memos, reports, and phone calls that might measure environmental scanning activity. She's been finding that the firms in both industries do about the same amount of scanning (no significant differences). However, in the midst of her data collection, Norton announces a merger with Symantec (software). A year later, Jean compares the amount of post-merger environmental scanning in the software industry with that in hardware industry. Just as Jean had hypothesized, a t-test shows a greater amount of post-merger scanning among the software firms. She concludes that her hypothesis was supported: mergers of competitors cause an increase in scanning.

Critique the study described above in terms of its design flaws. Identify the design Jean used. Briefly allude to problems the study has with external validity, but concentrate on the Cook and Campbell threats to internal validity. Provide a rationale for why each threat is or isn't operating in the study. Link each of the relevant threats to the specific constructs being studied (mergers and environmental scanning). Finally, do a summary evaluation -- what can be learned from this study?

H) Quasi-Experimental Design

H1. The University of Mississippi observed that the graduation rate of students with poor grades during their freshman year was much lower than those with better grades. Wanting to improve the chances of these low performing students they tested the effects of a new rule for freshmen. The rule stated the following: "At the end of their first semester all students with a GPA between 2.00 and 2.25 must register for EDUC 1003 during their second semester." Students with a GPA lower than 2.00 were put on academic probation. EDUC 1003 consisted of a course whose objective was to help students develop good study skills. The theory was that poor study skills were a significant cause of poor academic performance, and that study skills could be learned.

They wanted to evaluate the effectiveness of this intervention so they collected data on the GPA's of students beginning with their first semester and continuing through to their third semester. At that point they wanted to analyze the data to determine whether the new rule was making a significant difference in the students' academic performance.

What is the best way to analyze these data to reach an interpretable conclusion? Describe your approach and then discuss the major threats to validity that it has. Their sample size of students was as follows:

Total N= 3,454 (these are students who completed all 3 semesters)

Number of students with first semester grades between 2.00 and 2.25 (and thus took EDUC 1003) was 956

H2. A nationwide furniture company wants to determine if a "diversity training" program will improve individual sales performance on the floors of its urban retail outlets. A group of 45 salespeople in one city were given two weeks of training. Measures of individual sales performance were taken once two months prior to the program and once two months after. Forty-five salespeople (who did not receive the training) in a similar city had the same measures taken at approximately the same points in time. The training group increased their average weekly sales from $14,000 to $16,500, while the non-training group improved theirs from $13,000 to $14,500. The change for the training group is statistically significant; the change for the non-training group is not.

How would the interpretation of these results in this quasi-experimental design differ from a true experimental design? What are the threats to internal validity in the above study, and how might they be operating (be specific)? What additional information and analyses would be useful in controlling for these threats?

H3. An accounting firm is interested in determining if a re-training program will reduce errors made by its audit teams. A group of 45 auditors in the home office location are given two weeks of re-training. Measures of the frequency of detectable errors are taken once two months prior to the program and once two months after. Forty-five auditors in similar office at a branch location have similar measures taken at approximately the same points in time. The training group reduced their average number of errors per audit from 8 to 3, while the non-training group reduced theirs from 11 to 9. The change for the training group is statistically significant; the change for the non-training group is not.

How would the interpretation of these results in this quasi-experimental design differ from a true experimental design? What are the threats to internal validity in the above study, and how might they be operating (be specific)? What additional information and analyses would be useful in controlling for these threats?

H4. Describe the main classes of threats to validity that are found in quasi-experimental research designs.
H5. Dr. Delta, an organizational development consultant, was hired by a firm that has 600 pizza parlors nationwide to assist in implementing organizational changes that he hopes will result in higher levels of job involvement. At the outset of the study subjects had jobs with very low levels of enrichment and were paid on an hourly basis. He designed an experiment in which stores were randomly assigned to one of three treatment conditions: Condition 1 subjects experienced job enrichment (JE) and a change to performance-based pay (PBP); Condition 2 subjects experienced only JE; and Condition 3 subjects experienced only PBP. The parlors were sufficiently far apart from one another and communication among the parlors so poor that subjects in any parlor were totally unaware of changes made in other parlors. At the end of 18 months he measured the job involvement levels of workers and then tested for differences between the three conditions using a standard one-way analysis of variance. In view of Dr. Delta's design, answer the following questions:

How sound is Dr. Delta's design? Comment on the weaknesses of the design.

I) Interrupted Time Series Design

I1. What threats to internal validity are and are not ruled out by an interrupted time series design that includes the introduction of a treatment and its removal at a subsequent time period? List and explain. In the process, be certain that you provide a diagram that illustrates such a design.

J) Validity {Some Questions are about the validity of a measure; others about experimental validity)

J1. It was some time ago that the "tripartite" view of validity was abandoned in favor of a unitarian view in which different evidentiary bases for validity might be put forward. Binning & Barrett (1989) represent an attempt to place multiple evidentiary bases into a single framework. Cook & Campbell (1979) describe four aspects of validity, but they aren't really meant to represent evidentiary bases. Adding to the confusion is the fact that, while Binning & Barrett and Cook & Campbell use some of the same terms, these two pairs of authors clearly mean different things by these terms and intend for them to be used in different ways. Your task is to 1) Describe the evidentiary bases discussed by Binning & Barrett, 2) Describe the aspects of validity described by Cook & Campbell, 3) Describe the similarities and differences between the two lists including the different purposes served by the two lists, and 4) Present a single framework that incorporates both. This last task is the most important of the four.
J2. What are the types of validity? How would each affect an empirical study? IC
J3. Define both internal and external validity. Which is more important (it is not okay to say that they are equally important)? Defend your reasoning.
J4. Why might someone argue that the construct validity of a measure can never really be established?
J5. Describe each of the following "forms" of validity: content, convergent, discriminant, criterion-related, and construct. How is each assessed? How do they related to each other? Which is most important? (Make sure you defend your answer.)
J6. Define and explain 6 of the 8 types of validity listed below and describe the type(s) of information each provides. Choose one specific type of study as an example (e.g., cross-sectional study, experiment, quasi-experiment, simulation, ethnography, etc.) and compare and contrast the usefulness and importance of each type of validity for that type of study.

-- Concurrent validity

-- Construct validity

-- Content validity

-- Convergent validity

-- Discriminant validity

-- Statistical conclusion validity

-- Internal validity

-- External validity

J7. Designing empirical research has been called "making a series of trade-offs". Unfortunately, decisions that are made to increase one desired quality (like internal validity) can also decrease another desired quality (like external validity). For each of the issues below, discuss the concerns involved (i.e., the trade-off) and identify the qualities that are being increased or decreased.

a. laboratory vs. field study

b. using a 5-item measure vs. a 1-item measure

c. using a student sample vs. a sample of full-time employees

d. using a sample of employees from one company of 5000 vs. 500 employees from each of 10 companies

e. collecting your data using one method vs. collecting your data using multiple methods.

J8. There are four major types of validity in social science: internal validity, external validity, construct validity and statistical conclusion validity. Please address the following three issues concerning validity:

a. What is the meaning of the four major forms of validity (i.e., internal and external validity, construct validity, statistical conclusion validity)?

b. Are there trade-offs that can, or must, be made among the four types of validity?

d. Describe how the concept of validity varies across the research traditions of experimental vs. observational research.

IC
J9. The quality of research is often assessed by examining validity issues. As you are aware, there are numerous types/forms of validity (e.g., internal and external validity, construct validity, statistical conclusion validity). These types of validity address different aspects of the research process. For this question, please address the following issues:

a. What is the meaning of the four major forms of validity (i.e., internal and external validity, construct validity, statistical conclusion validity).

b. Should there be a priority among these forms of validity? If yes, why. If no, why not.

c. Does the concept of validity vary across research traditions (i.e., experimental vs. observational research). If yes, why. If no, why not.

J10. External validity has received considerable attention in the social and behavioral sciences. Some researchers argue we are too concerned with the external validity of our findings, while other researchers argue that we are too unconcerned with external validity. Several points of view have been developed to explain what is external validity. When answering this question, please be sure to address at least the following issues:

a. Compare and contrast alternative conceptualizations of external validity.

b. Some researchers have argued that the role of external validity is invariant across research traditions (i.e., experimental vs. observational research). Defend the perspective that the role external validity is invariant across these two research traditions. In addition, present the weaknesses of this perspective.

J11. In designing a research study, researchers must consider trade-offs between internal and external validity. Explain what is meant by internal validity and external validity. What is the nature of these trade-offs? How would you evaluate/determine which is more important, internal or external validity, for a particular study? Why is this such an important issue in organizational studies?
J12. A researcher conducted a quasi-experiment to test the effects of participative decision making (PDM) on job attitudes of employees. Employees from different departments in a large organization were assigned to either a treatment group (PDM implemented) or a control group (no PDM). Random assignment was not possible because of work schedules, coordination of tasks, etc. Thus, the design can be considered an untreated control group design with pretest and posttest and is represented as follows (where O=observation, X = treatment):

________________

O1 X O2

-------------------------

O1    O2

_________________

The results of the experiment are presented in the graph below. The dependent variable is job satisfaction and is measured on a 100 point scale (with 100 being extremely satisfied). How amenable are these results to causal interpretation? What is (are) the major threat(s) to internal validity B if any B for this type of design, given these results? What steps can be taken to assess the extent of this (these) threat(s)?

K) Data Collection Methods

K1. Within "micro" organizational research, much criticism has been raised concerning the reliance on self-report survey data. What potential problems does self-report survey research raise? How big of a problem do you think this is?
K2. There are a number of methods for collecting data (e.g., survey, interview, archival, etc.) that are available to a social scientist/organizational researcher. Discuss each in terms of their strengths and weaknesses, both on their own and in comparison to one another.
K3. Pick an important construct in your favorite area of management research for which there is some controversy about how it should be measured or operationalized (e.g., "environmental munificence" or "growth need strength"). Give the prevailing verbal definition of the construct, and analyze the measurement controversy from two broad angles. First, discuss how the controversy may or may not stem from claims about how one measure is better or worse than another because it (or the other) is an (a) archival record, (b) direct observation, (c) social report, or (d) self-report. That is, what are the relative strengths and weaknesses of each of these types of measurement for the construct you're discussing? Second, discuss the construct validity of the most commonly used measure(s) of this construct. To answer this second part you need to discuss just what construct validity is, and how it can or cannot be demonstrated. In the situation you describe, what could be done to demonstrate or improve the construct validity of existing measures?
K4. Interviewing, in some form, has been the workhorse for data collection in a variety of research traditions: positivist/quantitative, ethnographic, feminist/post-modernist, and, increasingly, historical research. While each tradition elicits information from respondents, the goals and processes involved in interviewing vary considerably across research traditions. Discuss the purposes of the interview, the relationship between data and theory, the roles assumed by the interviewer and the respondent, and the relevant criteria for assessing the validity and reliability of the information obtained. In addition, compare and contrast the use of the interview as a data collection method in the positivist/quantitative tradition with its use in one of the alternative traditions.

L) Common Method Variance

L1. Many of us have had concerns about the reviews of our survey research which suggest that data analyses of surveys are biased because of common method variance, effects of individual differences such as negative mood, negative affectivity, and social desirability, and the reliability and validity of the constructs we are attempting to measure. Define how these problems affect a research issue in organizational behavior and the steps that might be taken to minimize such potential criticisms of papers based on analyses of self-reported data through survey instruments.
L2. In a series of papers by Spector (1994), Schmitt (1994), and Howard (1994), published in the Journal of Organizational Behavior, arguments were provided for and against the use of self-report questionnaire measures. One of the issues discussed that could limit conclusions drawn from relationships observed using self-report measures was method variance. What is meant by method variance, what are its sources, and what impact is it expected to have on research findings?

M) Sampling

M1. The manufacturing census samples all plants of 250 employees or more. For plants of 5-249 employees it takes a size proportional sample. Plants of 1-4 employees are not sampled. What challenges does this sampling method provide to researchers? What could you do to account for these problems if you were studying organizational (not plant) growth? TH
M2. New York State has approximately 250,000 employees in over 70 state agencies, commissions and boards, ranging in size from approximately 50 to over 5,000 employees. Assume that you wanted to draw a representative sample of 1000 employees. Describe two ways that you could draw such a sample and briefly describe the advantages and disadvantages of each of these sampling approaches. IC
M3. The director of a national non-profit agency with approximately 10,000 employees, located primarily in regional offices in each of the 50 states the United States, has approached you and asked you to develop a study (organizational survey) of organizational culture within the agency. The agency is about to embark on a variety of change initiatives, including developing team structures and the study is meant to examine various elements of organizational culture that are potential barriers to change. The director has indicated that he would like to have the data analyzed at both the individual and office level of analysis. Because of funding limitations, the agency will only be able to send the survey to 1,000 employees. The director has indicated that there are three ways of sorting employees: all employees listed alphabetically, all employees listed alphabetically by state, and all employees listed alphabetically by region (there are between five and ten regions per state). Given these three ways of sorting employees, describe at least three different sampling approaches and the advantages and disadvantages of each.
M4. In your role as Teacher of Organizational Studies you are asked to prepare a tutorial for beginning doctoral students on sampling design. Assume that this tutorial is part of a basic class in research design and that students in this class have had a basic statistics course, up to and including multiple regression. Your tutorial should focus on briefly defining and explaining the differences between probability and non-probability samples. You should make sure to present and describe several types of probability samples (e.g., random, systematic, stratified, etc.) and non-probability samples (e.g., convenience, snow-ball, quota, etc.), and discuss general principles for when to use a probability vs. a non-probability sampling design. For each sampling design, you should also provide at least one clear example of a situation where this type of sampling design is appropriate.

N) Cross-Cultural Research

N1. Discuss the problems in conducting cross-cultural/national comparison in terms of (1) conceptualization, and (2) research design and data collection.

Define the three levels of measurement invariance. How would the problems that you have identified in part (a) affect measurement invariance? How would you examine measurement invariance by confirmatory factor analysis?

TH
N2. Explain why using national boundaries as independent variable in cross-cultural studies may not be appropriate in terms of theory and research methods. What alternative ways would you recommend? IC

O) General

O1. In your readings, research has been described as choices among conflicting goals (desiderata). For instance, when choosing among research strategies, a researcher must choose among the desiderata of precision, realism, and generalizability. The trade-offs among desiderata occur for all aspects of research; that is, for the selection of theories, the treatment of independent variables, research strategies, the selection of the problem to study, rules of evidence for determining the meaningfulness of a research finding, and even the frameworks that describe the research process.

Discuss what you see as the major trade-offs for each of the following aspects of research. That is, discuss what you see as the major strengths and weaknesses (advantages and disadvantages) of each aspect.

a. The trade-offs between internal and external validity.

b. The manipulation of the independent variable vs. the observation of the independent variable.

c. Basic vs. applied research.


3. DESIGNING A RESEARCH STUDY

A) General (Research Design)

A1. It has become almost a truism to note that the choice of methods influences theory and questions, data, and conclusions (the causal direction might also be reversed). Briefly discuss how the methods used have influenced research in any two of the following areas: leadership, power and politics, organization culture, organization justice. Compare and contrast as possible. Suggest some potentially fruitful methodological directions for future work in these same topical areas, and don't be shy about reflecting on broader issues of how the field of organizational behavior constructs its realities. Lastly, tie your discussion and reflections back to your own future as a researcher: what lessons or guidance do you derive from your understanding of these issues?
A2. Characterize as fully as possible what you see to be the methodological approach or approaches in strategic management research. Is there a dominant method or a wide variety? Discuss the reasons for the situation. What are the advantages, benefits etc. of the approach(es) in use? What assumptions (ontological, epistemological, ideological) do they embody? Are current efforts ideal or at least adequate? Are there other approaches that could be useful? If so, how might researchers be encouraged to use these as well? If you see no advantage to more variety in methodological approach, support that point of view.

A3. Describe a research question or area of interest to you, and why you think it is not only of interest but significance. Using your understanding of research methodology, describe two ways of pursing the question that are as divergent on as many dimensions as you can imagine. Describe the advantages and disadvantages of each approach. How strong a link or fit is there between a research question and a method?

A4. In his 1957 article entitled, "The two disciplines of scientific psychology," Cronbach discussed the assumptions, goals, and methods used in experimental methods and correlational methods in psychology.

a. According to Cronbach, what is the main difference between the two disciplines?

b. Discuss the strength and weaknesses of each discipline.

c. What, if anything, could be done to improve the situation?

B) Designing a Research Study

B1. Cook and Campbell describe the experiment as the research design with the greatest methodological rigor. However, given the constraints present in most organizations, I-O Psychologists cannot often conduct experiments in applied settings, and must settle for techniques with less control than experiments. Describe a study to investigate the relationship between participation in a supervisor training program and work performance which could realistically be conducted in an organization. If an experiment is not possible, describe how you would go about maximizing your ability to draw valid conclusions from your design.

B2. The XYZ Co. has recently developed a sexual harassment web-based training program. The developers claim the program will cut training time and costs and will be as effective or more effective than classroom-based sessions. They believe that this will be due to the increased availability of the program and the ability for participants to review program material on demand. Before the XYZ Co. developers launch this product they want you to determine if their claims have merit. Explain how you would design a study for this purpose.

B3. Research suggests that demographic diversity improves group creativity but reduces group efficiency. Some argue that demographic diversity is just a proxy for value and experience diversity, and that demographic diversity on its own is meaningless. Design a test to settle this argument.

TH
B4. E-mail has become a common form of communication in organizations and is used for both formal and informal communication. Because it has become so popular, some people complain that the frequent use of e-mail leads to miscommunication because they receive communications in e-mail that should be accomplished through face-to-face or telephone conversations, where there is an opportunity for interaction. You are interested in looking at this phenomenon. You are most interested in describing how e-mail is used for both formal and informal communication and in examining the effect of the e-mail use on organizational communication effectiveness. Design empirical research to address those issues.

Describe your study in detail, thoroughly explaining the research design, sampling plan, data collection procedures, ethical issues and proposed data analysis for your research design. Be sure to discuss the internal and external validity of your research. Which validity threats are ruled out and which remain?

REMEMBER: Be thorough. Your research design should demonstrate that you are a competent researcher, who is knowledgeable about the proposed research methodology or methodologies. Therefore, be sure that you carefully describe and give technical explanations for each component of your research

IC
B5. Choose a specific situation involving employee selection, e.g., artists for an advertising agency. (a) What strategy would you use in selecting the selection measures? (b) How would you combine the measures, e.g., which measures would you combine in a compensatory manner and which is a noncompensatory manner? (c) How would you assume test fairness, defining various measures of the term? (d) How would you make final selections? (e) How would you validate your procedure? (f) What considerations would enter in if your company were too small to provide validation data? (g) What difference would it make if you were developing a procedure for deciding whether trainees should be promoted as opposed to making new hires? Please note--emphasize, but do not limit yourself to statistical issues.

B6. There is evidence in the marketing literature of a non-monotonic relationship between the expectations that customers have for retail service and their eventual level of satisfaction with that service. That is, as a customer's expectations for service increase, so does their eventual service satisfaction -- up to a point. After that point, the greater the expectations the lower the eventual satisfaction. Develop a hypothesis that includes a moderator variable that might account for this evidence. Give some reasonable rationale for it. State your hypothesis verbally, as an equation, as a directed graph (circle & arrow diagram), and as an X-Y function chart. Discuss how it would or would not be possible or useful to test your hypothesis via a laboratory experiment. In your discussion, cover how and why a researcher could (not) manipulate the independent constructs involved (mode X), whether or not random assignment to conditions (mode R) is possible (including how may conditions would be necessary), which alternative constructs could or could not be controlled (mode K), and which constructs should be measured (mode Y).

B7. Assume you are the chairperson of a committee that must evaluate undergraduate programs at your university. Your task, as chairperson, is to devise an approach (a design, a research strategy, a set of measures) that will allow you to evaluate undergraduate programs. Make sure that your approach can be used in the future and at other comparable universities. When developing your plan, be sure to address at least the following issues:

a. the type of design you would select. Compare and contrast the selected design with another possible design.

b. the type of research strategy you would select. Compare and contrast with another possible strategies.

c. discuss how your design and approach deals with the major forms of validity. Be sure to examine at least internal, external, and construct validity.

B8. Assume you are an organizational research design expert assigned to the Strategic Planning Department of a large multi-national manufacturing firm. The Department has developed a new, PC-based, software package called "STRATPACK". The software (and a one-week training course to learn how to use it) is intended to improve the long-range planning ability of middle and upper-level managers.

Your job as research design expert is to devise research (a design, a research strategy, a set of measures) that will allow you to evaluate the effectiveness of the STRATPACK program. Describe a research design for evaluating the program. Make sure that your research approach can be used in the future and at all sites in the company. When developing your plan, be sure to address at least the following issues:

a. discuss how your design and approach deals with the major forms of validity. Be sure to examine at least internal, external, and construct validity.

b. the type of design you would select (e.g., a design where you manipulate the independent variable versus observe the independent variable). Compare and contrast the selected design with another possible design.

c. how your design deals with the major forms of validity. Be sure to examine at least internal, external, and construct validity

B9. Assume you are a consultant hired to evaluate the effectiveness of an industrial safety program undertaken by a large manufacturing organization. The safety program is a behavioral one that provides rewards such as monetary bonuses, or extra paid vacation days, for safety behaviors in terms of accident-free days, weeks, etc. Your task is to devise an approach (a research strategy, a design, a set of measures, and a general analysis strategy) that will allow you to evaluate the safety program. Because of the potential for the influence of many extraneous variables on any safety program, it is difficult to design research a single, cross-sectional study than can evaluate the impact of a single safety program. How would you design a study to evaluate the impact of the safety program? When developing your plan, be sure to address at least the following issues:

a. the type of research strategy you would select. Compare and contrast with another possible strategies.

b. the type of design you would select. Compare and contrast the selected design with another possible design.

c. discuss how your design and approach deals with the major forms of validity. Be sure to examine at least internal, external, and construct validity.

B10. One of the most important areas of research is forecasting of human resources needs based on strategic positions of organizations. For example, NY State must carefully forecast its human resources needs as it proceeds with significant lay-offs of state employees. Future human resources needs are generally evaluated by either statistical, i.e., regression or simulation, or clinical judgment, i. e., supervisors judgments, methods.

a. Design a single study that addresses the empirical question of whether statistical or judgmental methods are better for forecasting human resources needs. You may discuss a program of research, but please be quite specific in your description of the first study.

b. What are the critical methodological issues in doing this kind of research?

c. What are the critical applied (or practical) issues in doing this research?

d. Are the issues in b and c related?

B11. In the past decade, computing/information technology has changed many aspects of organizational life. For example, the technology can make increased decentralization possible by improving communication between central offices and field offices. Computer networks and word processors can enhance group empowerment by making it possible for team members to create and edit written group products quickly, even if the individuals are at different locations. Some have argued that computing/information technology will foster the paperless, productive office; nevertheless, others argue that the technology only increases paper and reduces office productivity. You are applying to the National Science Foundation for a grant to conduct research that examines the impact of computing/ information technology on organizational decision processes. The NSF application procedure imposes five basic requirements that your research design must fulfill in order to receive funding. Note: A very significant factor affecting NSF grant decisions is a clear demonstration that the researcher is knowledgeable about the research methodology used. Write your response to each of the following five grant requirements. Be sure not only to respond to the requirement, but also to demonstrate your research competence.

1.You must design a study that uses two different methods (interview/observation, survey, experiment, or quasi-experiment). Describe your two methods in detail and provide a rationale for the key elements (e.g., sampling, data collection, proposed analysis) of your methods. Be sure to discuss and evaluate the internal and external validity of both methods.

2. As part of your research design, you must select a conceptual (latent) variable central to your proposal (use "centralization of decision making" as your variable) and describe how you will develop your own measure. (You may not simply use "off the shelf" instruments).

3. You must describe how you will test for the reliability of the measure (centralization of decision making) you developed in step 2. The NSF insists that you describe how you will evaluate both the temporal stability and the internal consistency of your measure. Be sure to discuss what temporal stability and internal reliability mean and how they are related.

4.You must describe how you will assess the validity of the measure (centralization of decision making) you developed in step 2.

5. NSF requires a brief statement and defense of the philosophy of science underlying your research. Discuss your position and show how your research design is consistent with your philosophy.

B12. Many organizations in the United States economy have installed some form of Empowered Teams as an alternative, or supplement, to traditional bureaucratic structure. Preliminary evidence of the effectiveness of empowered teams, however, is mixed--some organizations have claimed success and others have reported failures. The evidence, however, is predominantly anecdotal--written for trade magazines by the organizations themselves. There are few rigorous, empirical studies assessing the effectiveness of empowered, or self-managed, teams. You are applying to the United States Department of Labor (USDOL) for a grant to conduct research that examines the effectiveness of empowered teams. The application procedure of the USDOL imposes three requirements that your research design must fulfill in order to receive funding. Write your response to each of the three grant requirements. The importance of each section is implied by the evaluation points allocated by the USDOL to each of the three requirements given in parentheses.

1. (50 points). Your study design for assessing Empowered Team effectiveness must use two different methods (interview, observation, archival analysis, survey, simulation, experiment, or quasi-experiment). Describe in detail and thoroughly explain the sampling plan, instrumentation, data collection procedures, and proposed data analyses for both methods. Be sure to discuss the internal and external validity of both methods, and how the two methods complement one another.

2. (30 points). As part of your research design, you must select one conceptual (latent) variable central to your proposal and describe how you will develop your own operationalization of the measure. (You may not simply use "off the shelf" instruments). Use the concept of "group morale" to demonstrate your operationalization methodology. Be sure to explain very explicitly to the USDOL how you will assess the validity and reliability of the "group morale" measure you are developing.

3. (20 points). The USDOL traditionally has funded research conducted only within the general paradigm of logical positivism, but now is more open to other research paradigms as well. Your grant proposal must provide a statement and defense of the explicit philosophy of science underlying your research. Describe your philosophy of science and show how your research design is consistent with your philosophy.

REMEMBER: Be thorough. You not only must design effective research, but you also must demonstrate to the USDOL that you are a competent researcher. It is absolutely vital to the USDOL that the principal researcher is knowledgeable about the proposed research methodologies. Therefore, be sure that you always carefully describe and give technical explanations for each component of your research design, and explain all technical terms (e.g., Cronbach alpha; KR-20; Likert scale).

B13. Assume that you are working as part of a research team that is responding to a RFP (Request for Proposals) from a funding agency for research on work socialization and newcomer adjustment. The RFP calls for rigorous field research that tests the effectiveness of theory-based interventions aimed at facilitating the adjustment of new organizational members to complex and/or stressful work conditions. The major goal of your team's research is to test the effectiveness of a self-regulation intervention, which trains newcomers to identify and use relevant social information and cues to regulate work behavior. Your group hypothesizes that self-regulation training will be more effective than other common treatments or interventions (e.g., realistic job previews, institutionalized socialization tactics).

You have been included in the team because of your methodological expertise. Your role is to design a rigorous field experiment or quasi-experiment to test he effectiveness of the self-regulation training program. REMEMBER, THE MAJOR CRITERION FOR FUNDING IS THE RIGOR OF THE RESEARCH DESIGN. Design a field study to include in the grant proposal. Your proposal should contain the following two sections:

1. Study Design: Construct and describe in detail an experimental or quasi-experimental design to test the effectiveness of your intervention. Describe in detail the experimental groups (including how they will be constructed), the research setting, the sampling plan for study participants, the treatment manipulation, and the data collection procedures. Also, indicate any control or nuisance variables that may be measured.

After describing the procedures, write a justification of the design. What threats to validity does it rule out or limit? and what threats to validity remain?

II. Measurement: Describe the steps you would take to achieve valid and reliable measure of the latent (conceptual) variable worker adjustment. Your team feels that as part of the proposal you must develop an original operationalization of adjustment (thus, you cannot simply use existing, "off the shelf" measures). Indicate in adequate detail how you would operationalize the measure, and indicate how you would demonstrate reliability and validity.

B14. In describing their notion of the collective mind Karl E. Weick and Karlene H. Roberts note: "Our focus is at once on individuals and the collective, since only individuals can contribute to a collective mind, but a collective mind is distinct from an individual because it inheres in the pattern of interrelated activities among many people." (p. 360) Collective mind is not indexed by within-group similarity of attitudes, ideas, or activities. A collective mind is one in which many people or many groups integrate their different attitudes, ideas and activities heedfully. As Weick and Roberts note, "People act heedfully when they act more or less carefully, critically, consistently, purposefully, attentively, studiously, vigilantly, conscientiously, pertinaciously,,, "(p. 361)

The idea of the collective mind as heedful interrelating is developed in the following excerpt: "When people make efforts to interrelate, these efforts can range from heedful to heedless. The more heed reflected in a pattern of interrelations, the more developed the collective mind and the greater the capability to comprehend unexpected events that evolve rapidly in unexpected ways. When we say that a collective mind 'comprehends' unexpected events, we mean that heedful interrelating connects sufficient know-how to meet situational demands. For organizations concerned with reliability, those developments often consist of unexpected, non-sequential interaction..." (p.366)

An example of heedful interrelating occurs as a pilot lands her/his aircraft on the deck of an aircraft carrier. The pilot does not actually land. S/he is "recovered." A recovery is an interrelated set of activities among air traffic controllers, landing signal officers, the control tower, navigators, deck hands, the helmsman driving the ship, etc. These activities can be interrelated more or less heedfully, depending on the care with which the different units contribute, represent the situation, and subordinate their activities to the single purpose of successful recovery. (p. 363)

You are to design a two-part study.

Part I. You are to design a measurement study which assesses heedful interrelating (i.e., collective mind) with a quantitative measure in a replication of an aircraft carrier flight deck or other situation of your choosing. Describe your research strategies including operationalization of the variable, selection of the sample subjects, data collection, data analysis, and so on for the purpose of developing a measure of heedful interrelating.

Part II. Using the measurement tool developed in Part I, and any other tools as needed, describe how you would design a study which would compare heedful interrelating across organizations. Choose at least one macro-organizational variable (e.g., technology, environment) as your basis of comparison. Here you must describe how you would make your selection of organizations to be compared, your data collection procedure, data analysis, and so on.

B15. Many organizations in the United States have installed cross-functional teams to counteract the "silo mentality" of traditional bureaucratic structure. Preliminary evidence of the effectiveness of cross-functional teams, however, is mixed --some organizations have claimed success and others have reported failures. The evidence, moreover, is predominantly anecdotal--written for trade magazines by the organizations themselves. Since there have been few rigorous, empirical field studies assessing the effectiveness of cross functional teams, the National Science Foundation has decided to fund research on the topic.

Assume that you are submitting a research proposal to NSF for a grant to research the effectiveness of cross-functional teams. Describe in detail and thoroughly explain the research design, sampling plan, data collection procedures, ethical issues and proposed data analysis for your research design. Be sure to discuss the internal and external validity of your research. Which validity threats are ruled out and which remain? NSF also requires that you provide a statement and defense of the explicit philosophy of science underlying your research. Describe your philosophy of science and discuss how your research design is consistent with your philosophy.

REMEMBER: Be thorough. You not only must design effective research, but you must also demonstrate to NSF that you are a competent researcher. It is absolutely vital to NSF that the principal researcher (you) is knowledgeable about the proposed research methodologies. Therefore, be sure that you always carefully describe and give technical explanations for each component of your research design, and explain all technical terms (e.g., Cronbach alpha; KR-20; Likert scale).

B16. Assume that you are submitting a research proposal to the Department of Commerce (DoC) for a grant to research the effectiveness of virtual work teams. A virtual work team is a leaderless collection of individuals who are geographically separate from one another and share their expertise using internet media such as a listserver or Lotus notes. The DoC requires that the researcher use both qualitative and quantitative methods to gather data. Describe in detail and thoroughly explain the research design, sampling plan, data collection procedures, ethical issues and proposed data analysis for your research design. Be sure to discuss the internal and external validity of your research. Which validity threats are ruled out and which remain? The DoC also requires that you provide a statement and defense of the explicit philosophy of science underlying your research. Describe your philosophy of science and discuss how your research design is consistent with your philosophy.

REMEMBER: Be thorough. You not only must design effective research, but you must also demonstrate to DoC that you are a competent researcher. It is absolutely vital to DoC that the principal investigator (you) is knowledgeable about the proposed research methodologies. Therefore, be sure that you always carefully describe and give technical explanations for each component of your research.


4.

CRITIQUE OF AN EMPIRICAL ARTICLE


A1. In your role as Critic of organizational studies research you are to evaluate an article submitted to a journal and to describe in detail the weaknesses and the strengths of the research. Critique the attached article "An Exploratory, Hypothetico-Deductive Test of Mood Affects on Pro-Social Behavior in a Variety of Organizations," by an anonymous author submitted to the Journal of Organizational Research. The format for your critique should list the strengths and weaknesses of the research and give a clear rationale for all your criticisms and suggestions. In addition, it is important to make specific recommendations for improving the research and the manuscript. You should explain to the author how and why you would redesign the research and reanalyze the data to improve it. Finally, indicate your acceptance or rejection of the manuscript for publication. Be sure to explain and justify your criticisms and recommendations. In your review be sure to discuss:

1. The appropriateness and adequacy of the literature review.

2. Importance of the problem.

3. Conceptual development and support of the hypotheses

4. Appropriateness of the design for addressing the purpose of the study.

5. Soundness of research methods.

6. Adequacy of data analysis.

7. Legitimacy of the data interpretation and conclusions.

8. Quality of the discussion.

A2. Please evaluate the internal, external, construct, and statistical conclusion validity of [some recent article]. Provide an argument, given the current state of the literature on this topic, for which validity (or set of validities) is most critical at this juncture in the field. Finally, given your critique and assessment, design a follow-up study that improves upon the key validity (or validities).


5. MEASUREMENT

A) Measurement Terms

A1. There are four levels of measurement rigor in management research. Discuss each one. What properties do measurements at each level have (i.e., how much information do the numbers from each level of measurement convey)? How can the numbers be changed (transformed) and still retain the properties of their measurement level? Pick one construct from a favorite research domain and give an example of an operationalization of that construct at each of the four levels of measurement rigor (provide arguments about why you think those examples reflect each level). Lastly, treat each example/level as a dependent variable in some statistical analysis. Given an example of a statistical technique appropriate for analyzing data at that level of measurement rigor.
A2. Define (and discuss important features or issues concerning) the following constructs measurement. (Define any six terms).
1. Construct validity 6. Item analysis
2. Likert scaling 7. Semantic differential
3. Cronbach's alpha 8. True score
4. Unobtrusive ("trace") measurement 9. Generalizability theory
5. Interval scale 10. Criterion-related validity

11. Sociometric choice measures

B) Operationalization

B1. a) You are conducting a study which requires you to measure Hong Kong real estate agents' self-assessment of performance. Since there is no existing measurement available, you have to develop the instrument by yourself. Describe how you would develop the measurement assuming that you have no idea of how many dimension the construct has.

b) Suppose you have developed an instrument consists of 20 questions measuring 4 dimensions (5 items per dimension) from (a). You collected data from another 300 real estate agents to validate your instrument. Explain how you would use structural equations model (SEM) to examine the reliability and validity of the instrument. What are other concerns about reliability and validity that are not addressed by this approach?

TH
B2. The research and business world have been fascinated with the concept of organizational culture. However, operationalizing this construct has been a difficult task. Discuss the ways in which the construct has been operationalized (e.g., technique, static v. dynamic, sample selection, generalizability). Critically analyze each of these methodologies. Then choose a methodology you would use to examine organizational culture and defend your choice. Describe how you would study culture, and what more would be known about culture as a result of your study.

B3. Suppose you are setting out to develop a new measure of a construct. You want this measure to be used in survey research and you want to establish its validity and reliability. Your goal is to have at the end of the process a measure that is defensible and likely to be widely accepted by other researchers studying that construct. What steps would you take to conduct such a program of research? For each step, explain the purpose of that step, explain why that step is important, explain what methods and statistical procedures you would employ at each step, and explain what evidence you would hope to gather at each step.
B4. Assume you are interested in constructing a new measure of (fill in with a relevant variable from your field). Detail the major concerns that need to be addressed and the steps that need to be taken in working toward validating this new measure.
B5. Assume that you recently read an article on workplace violence, which you found very interesting. The authors argued that one of the predictors of workplace violence was a construct called "propensity to be violent", and used a one-item measure as follows: "How likely are you to become violent when provoked?" You had serious doubts about the authors' measure and decided to develop your own.

(a) How would you go about developing items to measure "propensity to be violent"?

(b) How would you determine whether the instrument was reliable?

How would you determine whether the instrument was valid?

B6. A concern in organizations is retirement of "baby boomers" within the next 5 - 10 years. This concern is focused on the loss of older employees with skills critical to the core business functions of organizations. In order to continue to have employees with those critical skills, it seems necessary to retrain some older workers. However, past practice in most organizations has been not to retrain older workers for a variety of reasons, mostly financial. A researcher working for the Human Resources Department of a large corporation - 135,00 employees in 12 different plant locations - hypothesizes that the major reason for not retraining older workers is ingrained in the culture of the corporation. The researcher feels that older workers are not as highly valued as younger workers in terms of their potential usefulness after retraining. The researcher searched the literature, but found nothing in the existing literature to measure the cultural value construct. What should the researcher do to establish a reliable and valid, self-administered paper and pencil measure of this construct, namely, the value of retraining older workers?

a. Please describe and explain the steps, in temporal order, that the researcher must follow to develop a measure.

b. How would you evaluate the measure based on the technical requirements of reliable and valid measurement as well as any practical issues in using the measure considering the size of the employee population?

IC
B7. Assume that you are going to conduct empirical research to assess the role that the complexity of an organization's manufacturing technology plays in determining the level of centrality of decision making in the organization. Describe and justify the steps you might follow to develop an operationalization of the concept of "technological complexity". What standards would you use to determine if your operationalization is valid?

B8. Organizational researchers have several strategies available to them for gathering information (e.g., self-report measures, archival measures, direct observation, and trace measures). Assume that you are going to conduct empirical research to assess the role that the structure of the organization plays in determining the level of organizational effectiveness. Briefly review the strengths and weaknesses of the basic strategies (that is, self-report measures, archival measures, direct observation, and trace measures) for measuring "organizational effectiveness". Next, describe and justify the steps you would take to develop an operationalization of the concept of "organizational effectiveness." What standards and procedures would you use to determine if your operationalization is valid?
B9. One of the most important topics of recent research has been stress in the work place. One of the most important problems with understanding this literature is the varying definitions and measurement of this construct. Suppose you had decided to investigate the relationship between stress at the work place and dual career family happiness.

a. Since there are multiple ways to measure stress in the work place, describe two ways to do this. Hint - avoid using two self-report measures.

b. Describe how you would decide on the best single measure. In doing so, describe the steps you would take in evaluating your measure. Be sure to define your meaning of validity and reliability. It is important that your measure have all the necessary psychometric characteristics so that other scientist can trust the results of your research.

c. Be certain that you answer the question: "Validity for what?"

d. In order to understand the results of your study, you need to compare your measure of stress at work with other measures of this construct. How would you do that?

B10. Assume that you have been hired by a national survey research company, Shash, Inc., to write items that will be used in a national survey of attitudes toward capital punishment. There are a number of criteria that can be used to guide your item writing. What are they? Briefly describe five (5) such criteria.

B11. Assume that you are going to conduct empirical research to determine the impact of "interpersonal trust" on organizational decision making. Describe and justify the steps you might follow to develop an operationalization of the "interpersonal trust" concept.

B12. Assume that you are going to conduct empirical field research to assess the role that "employee empowerment" plays in determining the level of organizational commitment of employees. Describe and justify/explain the steps you would follow to develop a valid and reliable operationalization of the concept of "employee empowerment."

Explain the procedures and standards you would use to determine if your operationalization is valid and reliable.

How would you assess the construct validity of such a measure?

Discuss issues concerning the level of observation of your measure and level of analysis of the data.

B13. Assume you are interested in developing a new scale to measure construct X. Further assume that you have used previous literature and your own conceptual development to delineate 2 dimensions of X, specifically XF1 and XF2. Describe the steps you would go through in developing, testing, and evaluating this new scale. With regard to testing and evaluating, be sure to consider the role of factor analysis (both exploratory and confirmatory) and item analysis. Also you will want to make sure that your new multidimensional scale shows evidence of validity (several kinds) and reliability, so be sure to examine these issues and explain how specific information gathered from the above testing contributes to these concerns.

C) Reliability

C1. Describe and discuss the concept of reliability and all of its various forms, including their assessment. When is each form appropriate and what information is provided by each index?
C2. In establishing congruence between multiple measurements, we often compute indices of reliability and/or indices of agreement. These two types of indices address very different questions. Your task is to 1) Define reliability and agreement, 2) Explain when each is useful using examples in which one but not the other is appropriate, and 3) Briefly describe two reliability indices and two agreement indices.
C3. Explain reliability in terms of the classical test theory. What are the consequences of unreliable measures in hypothesis testing? What are the factors that may affect the reliability of a multi-item scale? IC
C4. Generalizability theory is used within I/O psychology more often today. However, most researchers continue to use Classical Test Theory conceptualizations of error to determine reliability. What are the implications of using classical test theory versus generalizability theory to assess reliability?
C5. In establishing congruence between measurements, we often compute indices of reliability and/or agreement. Define reliability and agreement. Explain the differences in reliability and agreement and describe under what conditions assessing reliability is more appropriate and under what conditions assessing agreement is more appropriate.

C6. Define measurement reliability in terms of a specific measurement theory (e.g., true test score theory, domain sampling theory, generalizability theory, IRT). Discuss the different approaches for estimating reliability and the contexts in which each might be appropriate. Use examples throughout.

C7. What are the consequences of unreliability in a measure?
C8. What are some of the conditions that might keep the internal consistency (e.g., alpha) of a given measure low? Is a low alpha always a sign of a "problematic" measure?
C9. Define reliability. Name some methods of calculating reliability. How well do these methods map up with the definition of reliability? What is the impact of the number of test items and sample size on Chronbach's alpha and why would this effect occur or not occur? (Note: Do NOT discuss inter-rater reliability).
C10. Schmidt and Hunter (1996) described several types of measures typically used in organizational research and identified the type(s) of measurement error affecting each of them in various situations. For each measure below, indicate the types of error and sources of error that are most likely to affect it. Then indicate what could be done to reduce errors of measurement and/or what can be done to control for errors statistically.

a. Supervisor ratings of subordinate's job performance

b. Self-ratings of one's own job performance

c. Self-reports of one's own job-related attitudes

d. Sales figures for a salesperson (or production quantity for a production worker)

d. Aptitude tests taken as part of the hiring process

C11. Suppose you had conducted research on the relationship between organizational commitment and rigidity of organizational structure in 200 firms. Your measure of rigidity was based on number of organizational authority levels and the degree to which written organizational rules existed and defined role behaviors for employees. Thus, an organization with a large number of organizational levels and specific written policies defining appropriate role behaviors for employees would score high on rigidity of organizational structure. After analyzing your data, you find that there is a statistically significant, negative correlation between commitment and rigidity, which is consistent with your hypothesis. However, the reliabilities for your two measures are:

(1) commitment = .23 (2) rigidity = .83.

How should you interpret your results? What could you do to fix this situation? Describe the specific steps (in order) you would follow to develop a more reliable measure of commitment.

C12. (a) Define the concept of reliability (of measures) in a clear, concise, and accurate manner. (b) Provide a formula that expresses the reliability of a measure in terms of two components of variance, i.e., total variance and the error variance.

D) Multi-level Assessment

D1. One of the major challenges for group-level research is the issue of aggregation of individual data to the group level.

(a) Under what conditions (e.g., types of data, number of raters) should the various interrater agreement and reliability methods be used? (Rwg, Kappa and Weighted Kappa, percent agreement, average pairwise correlation, ANOVA methods (interclass correlation coefficient, WABA)) What are the advantages and disadvantages of each of the methods?

(b) Explain the controversy between Frank Schmidt, Larry James, and others regarding the differences between interrater agreement and interrater reliability. Who is right? Do the differences matter in research practice?

Defend your answers.

(c) Review two articles from the organizational behavior literature that have had to deal with aggregation issues. Critique the methods the authors used to make aggregation decisions.

(d) Provide some advice to researchers regarding what they should do if interrater agreement and reliability indices indicate that it is not legitimate to aggregate their data.

(e) Provide some ideas for avoiding the need to aggregate in group-level research through creative research methods and measurement strategies.

D2. Cross-level/multi-level research is becoming more evident in the organizational research literature. This type of research raises many issues and presents many challenges in terms of designing the research, measuring variables, and collecting and analyzing the data. Assume that you are interested in aggregating data (e.g., from the individual to the group, or from the firm to the industry) for one or more variables. What are the major issues you will need to deal with?
D3. A researcher wants to measure organizational climate for continuous learning. Organizational climate is operationalized as a shared, consensual aspect of the organization -- the shared perception among employees that the organization values and supports continuous learning. Describe how you would measure organizational climate in this case. Specify (a) the level (i.e., individual, group, organization) at which the construct exists, (b) the level at which you would measure the construct, and (c) the procedures that you would use to establish that measurement is aligned with theory (i.e., the construct validity of the measure).

E) General (Measurement)

E1. Studies in organizational behavior research often rely on single-item measures of key constructs. The use of single-item measures has long been criticized, based on the premise that single-item measures are unreliable. However, Wanous (Wanous & Hudy, 2001; Wanous & Reichers, 1996; Wanous, Reichers, & Hudy, 1997) has recently argued that single-item measures can exhibit adequate reliability. Wanous based this conclusion on a procedure for estimating the reliability of single-item measures using the classic formula for correction for attenuation. Subsequent studies have yielded mixed results regarding the conclusions of Wanous (e.g., Loo & Kells, 1998; Nagy, 2002), and the veracity of the perspective advocated by Wanous remains in question. Summarize and assess the arguments for and against the use of single-item measures, and evaluate the procedure and evidence offered by Wanous regarding the reliability of single-item measures. Based on your assessment and evaluation, develop recommendations regarding the use of single-item measures in organizational behavior research.

Gardner, D. G., Cummings, L. L., Dunham, R. B., & Pierce, J. L. (1998). Single-item versus multiple-item measurement scales: An empirical comparison. Educational and Psychological Measurement, 58, 898-915.

Loo, R., & Kells, P. (1998). A caveat on using single-item measures. Employee Assistance Quarterly, 14, 75-80.

Nagy, M. S. (2002). Using a single-item approach to measure facet job satisfaction. Journal of Occupational and Organizational Psychology, 75, 77-86.

Wanous, J. P., & Hudy, M. J. (2001). Single-item reliability: A replication and extension. Organizational Research Methods, 4, 361-375.

Wanous, J. P., & Reichers, A. E. (1996). Estimating the reliability of a single-item measure. Psychological Reports, 78, 631-634.

Wanous, J. P., Reichers, A. E., & Hudy, M. J. (1997). Overall job satisfaction: How good are single-item measures? Journal of Applied Psychology, 82, 247-252.

TH
E2. Ipsative measures have become prevalent in organizational behavior research. Ipsative measures ask respondents to answer questions relative to one another, such that the scores for all questions sum to the same constant for all respondents. Examples of ipsative measures include rankings, Q-sorts, and the allocation of a fixed number of points (e.g., 100) across a set of response options. Ipsative measures may be contrasted with absolute measures, which ask respondents to answer questions independently from one another. Absolute measures are exemplified by rating scales using an interval or ratio numeric response format. The relative merits of ipsative and absolute measures have been debated for decades (e.g., Cornwell & Dunlap, 1994; Hicks, 1970; Johnson, Wood, & Blinkhorn, 1988; Saville & Wilson, 1991, 1992), yet the fundamental issues underlying this debate have not been adequately addressed in organizational behavior research. For example, measures of organizational culture and values are often ipsative, and some researchers advocate these measures over absolute measures (e.g., Meglino & Ravlin, 1987; O'Reilly, Chatman, & Caldwell, 1991; Ravlin & Meglino, 1987; see also Rokeach & Ball-Rokeach, 1989). Review the relative merits of absolute and ipsative measures. Based on your review, which form of measurement is most appropriate, and why? Do your conclusions depend on the construct being measured? If so, what types of constructs are best suited to ipsative measures, and what types are best suited to absolute measures? Next, apply your conclusions to the measurement of culture and values in organizations. What specific arguments have been advanced in favor of ipsative measures of culture and values? How sound are these arguments? Should ipsative measures be used in the study of organizational culture and values? If not, how can absolute measures of culture and values be developed that address the concerns voiced by advocates of ipsative measures? Provide a specific example of the type of measure you would propose.

Cornwell, J. M., & Dunlap, W. P. (1994). On the questionable soundness of factoring ipsative data: A response to Saville & Willson (1991). Journal of Occupational and Organizational Psychology, 67, 89-100.

Hicks, L. E. (1970). Some properties of ipsative, normative, and forced-choice normative measures. Psychological Bulletin, 74, 167-184.

Johnson, C. E., Wood, R., & Blinkhorn, S. F. (1988). Spuriouser and supriouser: The use of ipsative personality tests. Journal of Occupational Psychology, 61, 153-162.

Meglino, B. M., & Ravlin, E. C. (1998). Individual values in organizations: Concepts, controversies, and research. Journal of Management, 24, 351-389.

O'Reilly, C. A., Chatman, J. A., & Caldwell, D. F. (1991). People and organizational culture: A Q-sort approach to assessing person-

TH
E3. In organizational research, measures often consist of items that represent different facets of a concept. Although such measures are prevalent and, in some cases, are advocated (e.g., Law et al., 1998, 1999), they can present a variety of conceptual and methodological problems. Describe the specific problems associated with measures that use items representing different facets of a concept. Compare these problems with the purported advantages of such measures. Conclude by indicating whether and under what conditions such measures are useful for organizational research.

Law, K. S., Wong, C. S., & Mobley, W. H. (1998). Toward a taxonomy of multidimensional constructs. Academy of Management Review, 23, 741-755.

Law, K. S., & Wong, C. S. (1999). Multidimensional constructs in structural equation analysis: An illustration using the job perception and job satisfaction constructs. Journal of Management, 25, 143-160.

TH
E4. What is "domain sampling" and what are its implications for measurement of constructs? Compare and contrast the influence of domain sampling on the following:

a. Confirmatory factor analysis studies

b. Experimental research

c. Qualitative field research

d. Quantitative field research

E5. Suppose Roberta Von Eisenbrick is a researcher interested in the consequences of AFFECTIVE CONFLICT (AC) in decision-making teams. She gets access to a variety of "outcomes assessment" committees at each of 87 different schools in the Dallas Independent School District, and gains permission to tape-record two of their meetings (one in February and one in March). Such committees have been formed to develop quantitative ways to track pursuit of school goals and provide accountability to local and state governments. These are often divisive teams, with heated meetings, because what gets assessed is what usually gets rewarded. After both of these meetings, the committee members each

fill out a short questionnaire that contains 20 Likert-type items measuring how much AC occurred (e.g., "People in our group often get angry with each other"). The individual scores are summed across items, then averaged across team members, to come up with a group-level AC score for each of the 87 * 2 meetings. The same procedures are followed for a 5-item measure of Group Cohesiveness (GC), which is predicted to be reduced by higher levels of AC. Roberta also listens to every tape, and from her expert judgment, she rates each group's meetings on the same 20 items. Those 20 ratings are summed for each of the 87 * 2 meetings. Now, Roberta is interested in evaluating her measure. She wants it to have excellent psychometric properties, including reliability and construct validity.

a. What is reliability? Provide both a formal and informal definition. Connect that definition to the context given above.

b. How could you estimate the reliability of this new measure of AC? Cover at least three kinds of estimates and the sources of error each one assumes or is estimating. Pick one of those kinds of estimates and provide a rationale for using it.

c. What is construct validity? Discuss how the (un)reliability of the AC measure could affect the conclusions one would draw from a study involving it and GC. In your discussion, be sure to go over the connection between reliability and validity, and the connection between reliability and the probability of Type I or Type II statistical errors.

d. What is common method variance? How does it complicate the picture described in (c)? Be specific about how it might affect the statistical errors mentioned above.


6.

STATISTICAL AND DATA ANALYSIS

A) Statistical Terms

A1. The omitted variable problem is (or should be) widely known, and most estimation techniques are subject to this problem. Describe the problem and discuss its theoretical and statistical consequences. What do you plan to do to deal with this problem in your own research? Can one ever fully address this concern in a given research context? Why or why not?
A2. Describe and compare the information provided by each of the following: (a) the size of the difference between the mean scores (or the value of a given score in terms of a particular scale of measurement); (b) the level of the statistical significance obtained; (c) the size of the confidence interval surrounding some summary statistic; (d) the proportion of variance accounted for. What conclusions can be drawn from each of these statistics in interpreting results of research? How do they relate to the design of the research?
A3. The following eight terms come from standard statistical analyses. Pick six of the eight and define them. Discuss important features, or issues, concerning each term.
1. Main effect 5 Bonferroni correction
2. Interaction effect 6. Unstandardized regression coefficient
3. Type II error 7. Statistical power
4. Oblique rotation 8. Generalized (ordinary) least squares criterion

(Other possible terms :

1. Main effect 7. Orthogonal comparison

2. Interaction effect 8. Omega-squared

3. Biserial correlation 9. F-test

4. Varimax rotation 10. Degrees of freedom

5. Type I error 11. Covariance

6. Statistical power 12. Non-parametric statistics)

B) Statistical Significance Testing and Power

B1. Describe the following with respect to statistical power: a) define the concept, b) list the factors that affect power and describe how they affect power, c) describe the steps required to plan research with respect to power, taking care to mention how the factors that you just described are included in the plan, and d) describe how power is increased through the use of repeated measures.
B2. In recent years (and in not so recent years, for that matter) methodologists have criticized the fact that psychologists rely heavily on significance testing.  What are the essential elements of these critiques? What are some remedies to the problems engendered by significance testing, and why have these remedies not been widely adopted?
B3. (a) There is a debate among the Industrial/Organizational psychologists on whether we should abandon significance testing and report confidence intervals instead. Please summarize arguments from both sides and state your position on this issue.

(b) From time to time some researchers want to confirm null hypotheses in their studies. What are the concerns of accepting a null hypothesis? What precautions we can do to provide a stronger support for a null hypothesis?

TH
B4. Explain the recent controversy about significance testing. Critique arguments for each side of the issue. If you were a big-wig in the APA, what changes, if any, would you recommend in the policies regarding significance testing in APA journals? Support your answer.
B5. Discuss the debate regarding statistical significance testing in psychology. What are some of the criticisms of significance testing? What is your opinion on these criticisms and on significance testing in general? Support your argument.
B6. What is statistical power? Why should we, as researchers and consumers of research, be concerned with it? What are some consequences of low power? What are some consequences of high power? What affects the power level of a statistical test? What can be done to increase power in various research designs (experimental, quasi-experimental, and correlational)?
B7. There has been an accusation made that empirical researchers depend too heavily on significance levels to interpret research. The accusers argue, among other things, that researchers have inappropriately sanctified the p < .05 level of significance. Describe and critique the arguments for and against the use of statistical significance testing in empirical research. What information other than significance levels can be used to interpret the meaning of empirical research?

C) Statistical Conclusion Validity

C1. Assume you have hypothesized a relationship between variables X and Y, conducted a study to investigate this relationship, and found it to be statistically non significant. What are the various reasons that might explain why the X-Y relationship was not significant?
C2. What is statistical conclusion validity, and what factors affect it? Indicate the nature of the effect on statistical conclusion validity of each factor you mention. What can researchers do to improve statistical conclusion validity?
C3. In general, what can/should a researcher do to maximize the likelihood that his or her statistical conclusions will be valid. Indicate the nature of the effect (on statistical conclusion validity) of each factor that you mention.

D) Moderators/Mediators

D1. Moderation and mediation provide different ways to model third variable effects. (a) Please define both of these terms. (b) Sketch a conceptual model to show how each relates to main effects. (c) Using variables of your choice, please write hypotheses to reflect both. (d) Explain how each is tested analytically.
D2. Discuss moderators and mediators. Provide an example of how (1) a moderating variable influences the relationship between the independent and dependent variable, and how (2) a mediating variable influences the relationship between the independent and dependent variable. Describe how you would test for both moderation and mediation using regression.

D3. Many theories in organizational research predict that the effects of one variable on another are moderated by a third variable. Such moderator effects are typically tested by entering the product of two variables as a predictor, after controlling for the two variables that constitute the product. Several researchers have pointed out that this product is often highly correlated with the squares of the two constituent variables and have suggested that moderating effects may be tested after controlling not only for the two constituent variables, but also their squares (e.g., Cortina, 1993; Ganzach, 1998; Lubinski & Humphreys, 1990; MacCallum & Mar, 1995). Discuss the strengths and weaknesses of this approach, as compared to methods that use only the product term and its constituent variables. Conclude with recommendations as to when and under what conditions organizational researchers should test moderator effects with the squares of the constituent variables in the equation.

Cortina, J. M. (1993). Interaction, nonlinearity, and multicollinearity: Implications for multiple regression. Journal of Management, 19, 915-922.

Ganzach, Y. (1998). Nonlinearity, multicollinearity, and the probability of Type II error in detecting interaction. Journal of Management, 24, 615-622.

Lubinski, D., & Humphreys, L. G. (1990). Assessing spurious "moderator effects": Illustrated substantively with the hypothesized ("synergistic") relation between spatial and mathematical ability. Psychological Bulletin, 107, 385-393.

MacCallum, R. C., & Mar, C. M. (1995). Distinguishing between moderator and quadratic effects in multiple regression. Psychological Bulletin, 118, 405-421.

TH
D4. Assume you are interested in studying relationships amongst three variables, X, Y, and Z. Your first hypothesis posits Z as a mediator of the relationship between X and Y. Your second hypothesis posits Z as a moderator of the relationship between X and Y. Draw the figures depicting each of these hypothesized relationships and describe (make sure to write regression equations) how you would test each of these hypotheses.
D5. The amount of time and effort that the average person pours into his or her job has risen steadily over the last two decades, such that the average person now is working one more week per year than the average person in the mid 1970's. This steady increase has fueled research interest in the construct of Job Burnout (JB), which is an emotional and physical exhaustion associated with one's work. Organizational Stress (OS), the perception of greater demands than available resources in one's work environment, is thought to be one of the chief antecedents of JB. However, time spent under OS is thought to make its affects on JB accumulate or worsen. Work Role Withdrawal (WRW), the avoidance of responsibilities or the deliberate reduction of inputs to one's job, is believed to be the chief consequence of JB.

a. Formally state these ideas as verbal hypotheses, as equations, and as a complex directed graph (circle & arrow diagram).

b. Conceptually, what is a mediator? Define the term and identify the mediator in your hypotheses in (a). What elements or steps in a study design (not statistics) that would be helpful for detecting the mediating relationship that was proposed.

c. What is a moderator? What are some of the other names for it within various subdomains of management? Further demonstrate your understanding of moderator variables by picking which variable in (a) is the moderator. Show the moderating relationship in an X-Y (or multi-layered) chart. Again, what elements or steps in a study design (not statistics) would be helpful for detecting the moderating effect implied above?

E) Multiple Comparisons

E1. In your role as professional colleague you are called upon to advise another doctoral student about the appropriate use of multiple comparisons and post hoc tests in analysis of variance. Your student colleague already has been introduced to ANOVA and multiple regression. Specifically, you are to write out a description of how to test for differences among the different levels, or cells, of a categorical variable. In your description you should discuss, among other things:

a. the statistical and philosophical issues raised by post hoc analysis and "data snooping".

b. the different multiple comparison methods commonly used by researchers to test for differences among cell means, and

c. the advantages and disadvantages of each method.

Note: Your written description will be evaluated based on your demonstrated competence with the statistical method, not on the pedagogy of your explanation.

F) Applied Data Analysis

F1. Attached is the beginning of an article.  Your task is to write the results section by:

 

1.         making up and reporting plausible descriptive statistics that will help the reader understand the results of the statistical tests you report and the nature of your sample (Hint: this can be presented in the form of a table.);

 

2.         describing and defending your choice of appropriate statistical tests of the hypotheses and the steps that you would go through in carrying out these tests;

 

3.         making up and reporting results of these tests that are consistent with your descriptive statistics and with any information in the rest of the article about the results found (Hint: this can be presented in the form of a table.);

 

4.         describing how well these results provide support for the hypotheses.  You may want to note any relevant threats to validity here.

 

Be sure to state any assumptions you make.

F2. Assume that the Personnel Manager of a local firm, Ms. B. d'Flandres, hired you to analyze some data that her assistants had collected over the past several years. The data consist of (a) job performance ratings that were made in June of 1990, and (b) scores on 25 predictor variables (e.g., measures of education, job experience, job tenure, job aptitude, personality) that were measured when the individuals were hired (between April 1987 and May 1988). She had data from only the 110 subjects (60 males and 50 females) who were hired during the period and were still employed by the firm. One objective of your analysis (analyses) is to determine how well the performance criterion can be predicted by the predictors. What strategy would you use in analyzing the data? Describe/explain. (b) Is sample size an issue for your analyses? Explain? If it is, what will you do to deal with problems stemming from sample size? Assume that no additional data can be collected at this time! (c) Some preliminary analyses that you do reveal that five of the predictor variables correlate (zero-order basis) about .75 with one another. Will this influence the analyses that you use? Will it influence the validity of conclusions derived from your proposed analyses. (d) Ms. d'Flandres is interested in knowing if the predictor variables are equally effective in predicting the job success of males and females. What analysis or analyses would you do to deal with this issue?
F3. Assume that you have been provided with data on 12 conceptually distinct variables that are moderately correlated with one another at the empirical level. The data are from students in four different college majors, i.e., astronomy (A), biology (B), criminology (C), and demography (D). The 12 measured variables fall into the general categories of mental health (tension, emotional withdrawal, hallucinations, guilt feelings, blunted affect, and suspiciousness) and job satisfaction (work itself, supervision, co-workers, pay, promotion prospects, and working conditions). Your task is to determine if the individuals in the four college majors differ from one another on the 12 measured variables.

(a) What statistical technique(s) would you use to test for group differences? Explain. Be specific about the procedures(s) that you would use.

(b) What statistical estimate would you use to estimate the extent to which the groups differ from one another? Be specific.

F4. The following statistical table, published in a recent issue of Work & Occupations, was taken from an article discussing variables influencing resistance and citizenship behaviors in the workplace. Resistance behaviors include playing dumb, withholding effort, work avoidance, subverting particular managers, and sabotage. Citizenship behaviors include cooperation, engaging in peer training, use of insider knowledge to facilitate production, commitment to organizational goals, giving extra effort freely. Both measures are global constructs aggregated to the interval level and standardized.

The table reports the results of a series of ordinary least squares regression equations with standardized coefficients. Explain the meaning of the statistics presented. What do these statistics tell us generally about the relationship between resistance behaviors and anomie, workplace control strategies, and workplace characteristics? What do they tell us generally about the relationship between citizenship behavior and anomie, workplace control strategies, and workplace characteristics? Discuss the relative advantages/disadvantages of using standardized regression coefficients in the data presentation. The measurement levels for independent variables are as follows:

Anomie Index -standardized interval level measure tapping multiple dimensions. The concept is used by the author to refer to norm violations that workers perceive. Work settings scoring high on the anomie index variable would be ones where managements act capriciously and arbitrarily in dealing with workers and would not tend sufficiently to production duties--i.e., planning, providing supplies, etc.

Forms of Workplace Control -all are dichotomous (coded 1=present; 0=absent), with some overlap within cases. Bilateral control refers to cases where management allows subordinates input into decision making.

Job Characteristics: -skill and autonomy variables are interval level measures. The union presence variable is dichotomous (1=union; 0=no union).

F5. Researchers collected data from a sample of 350 companies on the following variables:

size (measured as number of employees)

Business Strategy (a nominal variable with 2 response options: low-cost provider or differentiator)

HR Strategy (a nominal variable with 2 response categories: High commitment or low commitment)

Return on Equity (ROE)

Their proposition is that a firm's HR strategy should match up with their business strategy. Specifically, they believe that investing in high commitment workforce practices is needed when a firm is trying to deploy a differentiation business strategy. Furthermore, they believe that such a matching is especially important as firms grow larger.

Develop specific statistical hypotheses from these propositions using the variables described above. Then construct a regression model of the form Y = B1 X1 + B2 X2 + . . that would test these hypotheses. Explain what statistical test you would examine to reach a conclusion about each hypothesis.

G) Univariate/Bivariate Statistical Procedures

G1. In your role as Teacher of Organizational Studies you are asked to prepare a tutorial for undergraduate organizational studies students on basic statistical analysis procedures. Your students already have had an elementary course in statistics (e.g., probability, the normal, binomial, and t distributions, significance testing, the logic of hypothesis testing, etc.), but have not studied basic data analysis procedures. Outline a tutorial for your students thoroughly describing and explaining one of the following statistical analysis techniques: One-way ANOVA, Pearson product-moment correlation, or the chi-square measure of association. (Your tutorial can be conceptual; it need not present any mathematical formulas unless you wish to). In your tutorial be sure to discuss (among other issues):

1. The type of "output" that results from applying the technique. How to interpret results from the technique.

2.The assumptions underlying the technique, and the consequences of failures in the assumptions.

(Note: Your tutorial will be evaluated on your demonstrated competence with the statistical method, not on the pedagogy of the tutorial).

H) Multivariate Statistical Analysis

H1. In your role as Teacher of Organizational Studies you are asked to prepare a tutorial for graduate organizational studies students on multivariate statistical analysis procedures. Your students already have had the basic course in bivariate statistics (e.g., t-test, F-test, ANOVA, correlation, etc.) and are beginning their study of multivariate data analysis techniques. Outline a tutorial for your graduate students thoroughly describing and explaining one of the following multivariate statistical analysis techniques: Multiple regression, factor analysis, discriminant analysis, canonical analysis, cluster analysis, or MANOVA. (Your tutorial can be conceptual; it need not present any mathematical formulas unless you wish to). In your tutorial be sure to discuss (among other issues):

1. The type of "output" that results from applying the technique. How to interpret results from the technique.

2. The assumptions underlying the technique, and the consequences of failures in the assumptions.

(Note: Your tutorial will be evaluated on your demonstrated competence with the statistical method, not on the pedagogy of the tutorial).

I) ANOVA

I1. ANOVA. Assume that you have conducted a laboratory experiment in which there were 3 levels of Factor A (a1, a2, a3) and two levels of Factor B (b1, b2). Assume also that there were 20 subjects in each cell.

(a) Prepare a standard analysis of variance (ANOVA) table that is appropriate for this design. Make certain that you specify all effects and the appropriate degrees of freedom for each effect. If you use any abbreviations in your table make certain that you specify their meaning.

(b) Assume that an initial ANOVA analysis found no evidence of an A X B interaction. What, if anything, would you do with the sum of squares for the interaction effect?

I2. A study examined the effects of gender and pre-training motivation on the transfer of training in terms of improved performance for 123 new industrial managers. The training was on supervisory skills, and the dependent variable was based on the appraisal of the supervisors' performance before the training and three months after the training. The performance appraisal data were collected for research purposes only, and resulted in a total transfer score from 1 to 25, with 1 being low. Pre-training motivation was based on a 12 item Likert scale with an alpha of .86. Two groups were identified for the pre-training motivation variable - the bottom 1/3 (low) and the top 1/3 (high). Table 1 contains the ANOVA results and Table 2 contains the cell means.

a. What experimental design was used in this study?

b. What assumptions are made in the ANOVA model?

c How would you interpret the results in Tables 1 and 2?

d What additional computation can be done to evaluate the importance of the different effects in Table 1?

e Using the results in Table 2, show the G x M effect in a graph. (It is OK to draw this graph on a separate sheet and turn it in with your exam.)

f. Explain what type I and type II error mean in the context of this study.

Table 1: ANOVA Results for the Transfer of Training Study



Source of Variation SS df MS F



Gender (G) 249742.63 1 249742.63 2.35

Pre-training motivation (M) 426959.79 1 426959.79 4.02*

G x M 1257777.02 1 1257777.02 11.84***

Residual 12642667.00 119 106240.90



Totals 14577146.44 122

_________________________________________________________________

*p< .05; ***p< .001



Table 2: Cells Means



Pre-training Motivation

Gender Low High

Male 13.5 21.6

Female 15.7 18.3

J) Meta-analysis

J1. There exist two categories of literature review, narrative, and quantitative. Each category as typically implemented has certain advantages and disadvantages. Describe the advantages and disadvantages associated with each approach, taking particular care to enumerate the disadvantages associated with quantitative review. Then, offer two ways that meta-analysis might be combined with other analytical techniques to move beyond the focus of bivariate relationships that is typical of meta-analysis.
J2. Schmidt has argued that primary research does not allow one to discover general scientific laws, but that meta-analytic research does.  What are the arguments in favor of or against this position?  In what areas of research (if any) has Schmidt's prediction come closest to coming true?
J3. The so-called (by Hunter, Schmidt, and Jackson, 1983) "state of the art" meta-analysis method attempts to account for various sources of error that might produce apparent variation in effect sizes across studies. (a) Discuss the three most important sources of error that meta-analysis addresses. (b) Discuss the potential contribution that a meta-analysis can make to theory building in the organizational sciences. (c) Describe the circumstances that you would consider when deciding whether to perform a meta-analysis.
J4. Meta-analysis has been hailed as a "miracle cure" and criticized as "smoke and mirrors." Explain both sides of the debate and then lay out the role of meta-analysis as you see it within the time span of your career as an academic. To do this, use examples of already written (or yet to be written) articles as models.
J5. A frequent criticism of non-quantitative research summaries (literature reviews) is that researchers do not carefully examine how particular research designs might have affected the statistical outcomes of the studies included in the literature review. Meta-analysis, however, allows one to statistically examine such effects. Assume you are conducting a meta-analysis of the relationship between leader behaviors and group productivity. Briefly describe the steps you would take in conducting a meta-analysis. What aspects of the original research designs would you examine in your meta-analysis? What steps would you take in gathering and coding data to enhance the credibility of your analysis?

K) MTMM

K1. One method often used by social scientists to demonstrate the construct validity of measures is the multi-trait, multi-method matrix developed by Campbell and Fiske (1959). Briefly describe how this matrix is obtained or constructed by a researcher and then describe how each of the elements of the matrix is used to demonstrate the reliability and/or construct validity of the measure(s). IC
K2. Describe the logic of the multi-trait multi-method (MTMM) analysis in terms of the criteria for evaluating the validity of a set of measurement procedures. Interpret the following MTMM in terms of what it might infer about the validity of the different measurement procedures. You may impose any assumptions on the data you prefer. For example, you might want to discuss the matrix under the assumption that traits A, B, & C are mutually independent. Please justify and make explicit any assumptions you make.

Method I Method II Method III

____________ ____________ ____________

Trait A1 B1 C1 A2 B2 C2 A3 B3 C3



A1 (.89)

Method I B1 .51 (.89)

C1 .38 .37 (.76)



A2 .57 .22 .09 (.93)

Method II B2 .22 .57 .10 .68 (.94)

C2 .11 .11 .46 .59 .58 (.84)



A3 .56 .22 .11 .67 .42 .33 (.94)

Method III B3 .23 .58 .12 .43 .66 .34 .67 (.92)

C3 .11 .11 .45 .34 .32 .58 .58 .60 (.85)



(Reliabilities are in parentheses on the diagonal)

L) Multiple Regression

L1. Categorical variables can be analyzed in regression through the use of design variables. Three most common design variables are dummy codes, effect codes, and contrast codes. 1) Define, compare, and contrast these approaches to coding, 2) offer and describe substantive examples to which each approach is uniquely suited and 3) describe how interactions among categorical variables are examined in regression.
L2. Categorical variables can be analyzed in regression using design variables. The three most common design variables are dummy codes, effect codes, and contrast codes. Define, compare and contrast these approaches to coding, and describe what information is provided by the beta weights in each approach.

L3. Organizational researchers often encounter the problem of multicollinearity in their data analyses. Answer the following questions about this topic:



(a) What is multicollinearity?

(b) What are the sources of multicollinearity?

(c) What are the effects of multicollinearity on research?

(d) What are some diagnostics that researchers can use to detect multicollinearity?

What are some methods for dealing with multicollinearity? Discuss the adequacies of each method described.

L4. Various procedures can be used in carrying out a multiple regression analysis. Describe the reasons or conditions under which each of the five procedures listed below would be an appropriate analytic strategy. In addition, discuss how you would interpret the results yielded by the different procedures.

a. Simultaneous multiple regression

b. Hierarchical multiple regression

c. Stepwise multiple regression - forward

d. Stepwise multiple regression - backwards

e. Moderated multiple regression

IC
L5. Multiple regression is probably the statistical analysis procedure most commonly used in social science. Assume that you have a junior colleague who has had a basic course in bivariate statistics, but never has studied multiple regression. Write a brief tutorial for your colleague describing and explaining multiple regression. (Your tutorial can be conceptual; it need not present any mathematical formulas unless you wish to).
L6. Assume the following regression model: Y ' = b0 + b1x1 + b2x2 + b3x1 x2

Assume also that the Y' values are to be estimated using standard score (Z) data.



(a) What does b2 represent? Explain in a clear and concise manner.

(b) What does b3 represent? Explain in a clear and concise manner.

(c) What are the two most important determinants of the statistical significance of any of the ?i values? Specify and indicate how each is related to statistical significance.

L7. Multiple regression is probably the single most frequently used multivariate data analysis procedure in social science. It is imperative to understand the methods, assumptions, interpretation, and strengths/weaknesses of that oft-used data analysis methodology.

Define and discuss important features or issues concerning the following aspects of multiple regression.



Outputs of Regression Analysis



? F-test of significance and its associated degrees of freedom:

? Regression coefficient ("Beta weight"), standardized and/or non-standardized:

? R2 : The proportion of the total sum of squares that is accounted for by the predictors (linear and with non-linear terms)

? Shrunken, or adjusted, R2: (a downward adjustment in the population estimate of R2 based on its estimation in a sample of size N)

? Outliers:



Assumptions and Limitations of the Method



? Assumptions (describe/discuss the assumptions underlying multiple regression): (Normality and equality of variance, independence of the observations, linearity, interval-level measurement of all variables)

? Multi-colinearity:

? The least-squares criterion:



Methods of Regression



? Step-wise regression:

? Non-linear regression:

? Moderated regression:

M) General Linear Model

M1. The general linear model underlies most multivariate statistical methods. Describe and discuss the assumptions of the general linear model. Multiple regression, ANOVA, and principle components (factor) analysis are based on the general linear model. For each of those three statistical procedures describe and discuss the following:

a. The type of data necessary for using the technique appropriately.

b. The type of "output" that results from applying the technique.

c. The relative advantages and disadvantages of each technique.

d. The nature of the "dependent" and "independent" variable in each technique.

M2. Regression and ANOVA are based on the general linear model. Describe the general linear model and discuss its assumptions. For both regression and ANOVA discuss the following:

a. The assumptions of the techniques and the type of data necessary for using the technique properly

b. The output that results from applying the technique. How to interpret results.

c. The relative advantages and disadvantages of each technique

N) Clustering

N1. A recurring measurement issue is the assignment of objects to groups (categories/dimensions) based upon their similarities. Describe analytic techniques for performing this analysis. How could you assess the similarity of the objects? How could you assign similar objects to groups (categories/dimensions)? When answering this question, be sure to address at least the following issues:

a. The type of data necessary for the use of each technique,

b. The type of "output" that results from the application of the technique,

c. The criteria (and procedure) for determining the number of factors or groups,

d. The relative advantages and disadvantages of the technique.

O) Exploratory Factor Analysis and Confirmatory Factor Analysis

O1. Contrast Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA). Give an example for the appropriate use of each of the above analyses. What is the role of theory in the two factor analyses? IC
O2. Compare and contrast Exploratory Factor Analysis (EFA) with Confirmatory Factor Analysis (CFA). Put more emphasis on their theoretical basis, assumptions, usage and limitations. TH
O3. How do exploratory and confirmatory factor analysis differ? What are the advantages and disadvantages of each?
O4. Describe the differences between exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). When would you use one verses another? Are there ways to use EFA in a confirmatory way? Can you use CFA in an exploratory way? Also, describe the difference in principal components analysis (PCA) and EFA.
O5. There has been a remarkable convergence of evidence in the personality (individual differences) literature in the last ten years, toward the "Big Five" theory of personality structure. So, if a management researcher is interested in the enduring traits of top managers, job candidates, boundary role spanners, or any set of individuals, it is now requisite to measure one or more of the "Big Five" characteristics: Openness to Experience (O), Conscientiousness (C), Extraversion (E), Agreeableness/Conformity (A), and Neuroticism/Negative Affectivity (N). Unfortunately, almost all of the well-established measures of these personality dimensions are copyrighted and cost $1 or more per subject to use in research. Hence, there is a clear need for someone to develop a public domain set of measures of the "Big Five."

Suppose you were on a research team to develop that set of measures, and you're aiming for having 100 total items, with 20 in each dimensional subscale. Suppose also that you collect data using these new items from 350 undergraduates who also give your scale to their best friend, so that you can have both self- and social-reports of the undergraduates' personalities (350 people being rated, with 100 self-report items and another 100 social report items = 200 items total). How would you use exploratory and confirmatory factor analysis (EFA and CFA) to examine and help to validate your public domain measures? Specifically,



a -d. What steps would you go through, and what choices would you make in performing an EFA on this data set? In your answer, make some justified guess at the factor structure you expect to find.



Armed with your EFA results, how would you use (sketch out a plan without getting into great detail regarding degrees of freedom and so on) CFA to provide a more formal test of the fidelity of your measure? In your answer, distinguish the important differences in purposes, assumptions, and mechanics of the two types of techniques. How does hypothesis testing differ in this context relative to others, such as regression?

O6. The basic factor analysis model can be represented by the matrix algebraic equation: A B = C

(1) What do each of these matrices represent? Be specific as to the type of information in each matrix and its purpose.

(2) For what purpose do researchers use factor analysis? Explain. In the process, provide a very brief description of a study for which factor analysis would be a useful data analytic technique.

O7. Factor analysis is an important statistical technique in organizational research. Describe factor analysis as an analytical tool including the following issues:

a. the reasons for using factor analysis

b. the type of data necessary for using the technique appropriately.

c. the type of quantitative "output" that results from applying the technique

d. how a researcher would use the "output" in his or her research

e. how confirmatory factor analysis is different from exploratory

f. the important issues and problems in using the technique

P) Structural Equation Models

P1. Structural equation modeling (SEM) has become one of the more popular data analytic techniques in the social sciences. Like any analytic technique, it has its advantages and disadvantages. 1) Explain the advantage of SEM over regression-based path analysis associated with measurement error; 2) explain the notion of "maximum likelihood" estimation, broadly defined; 3) define, compare, and contrast any three of the fit indices offered by LISREL; and 4) discuss "unmeasured variables" problem ala James (JAP, 1980) and how it affects parameter estimation.
P2. There now exist dozens of model fit for SEM. Tanaka (in Bollen & Long, 1993, Testing structure Equation Models) offered six dimensions on which fit indices vary. The four dimensions with which you are concerned are Simplicity vs. Complexity, Normed vs. Nonnormed, Absolute vs. Relative, and Sample size independent vs. Size dependent. Your tasks are to explain these four distinctions and to give examples of fit indices that represent each category (e.g., Normed fit index and nonnormed fit index) explaining briefly why each index is representative of the category.
P3. An important consideration in structural equation modeling is whether manifest variables should consist of individual items or item parcels (i.e., sums or averages of individual items). For instance, a construct measured with 10 items may be specified by treating each item as a separate manifest variable or by averaging pairs of items, thereby yielding five manifest variables. The use of individual items versus item parcels has generated considerable controversy (e.g., Bandalos, 2002; Bandalos & Finney, 2001; Landis, Beal, & Tesluk, 2000; Marsh, Hau, Balla, & Grayson, 1998). Some researchers advocate the use of item parcels, whereas other research argue in favor of using individual items. Discuss the relative merits of using individual items and item parcels in structural equation modeling. Summarize your discussion by presenting guidelines that stipulate when and under what conditions organizational researchers should use individual items versus item parcels.

Bandalos, D. L. (2002). The effects of item parceling on goodness-of-fit and parameter estimate bias in structural equation modeling. Structural Equation Modeling, 9, 78-102.



Bandalos, D. L., & Finney, S. J. (2001). Item parceling issues in structural equation modeling. In G. A. Marcoulides & R. E. Schumacker (Eds.), New developments and techniques in structural equation modeling (pp. 269-296). Hillsdale, NJ: Erlbaum.

Landis, R. S., Beal, D. J., & Tesluk, P. E. (2000). A comparison of approaches to forming composite measures in structural equation models. Organizational Research Methods, 3, 186-207.

Marsh, H. W., Hau, K. T., Balla, J. R., & Grayson, D. (1998). Is more ever too much? The number of indicators per factor in confirmatory factor analysis. Multivariate Behavioral Research, 33, 181-220.

TH
P4. Some researchers argue that Structural Equation Modeling (SEM; e.g., LISREL) is a very powerful data analytic technique which has not been applied enough in substantive research while others would argue that its benefits have been greatly exaggerated and that it has been overapplied. Consider these perspectives by discussing the various strengths and weaknesses of SEM. In your discussion you should address, among other things, how SEM compares to other data analytic techniques. What do you think about the underapplication/overapplication of SEM?
P5. Structural equation modeling has gone from an obscure and complicated statistical tool to one that is commonly demanded by reviewers and used by relative novices. As SEM becomes more "user friendly" a number of statistical experts have voiced concerns about the wisdom of making programs like LISREL and EQS accessible to the "masses" (with simplified programming languages and graphic user interfaces).

Please comment on these restrictions versus access argument. Which side do you take?

What are the strengths of SEM? Give some examples.

What are the weaknesses of SEM? Give some examples.

MANY THANKS TO OUR CONTRIBUTORS

Arthur G. Bedeian; Kristin Byron; Wendy Casper; Gilad Chan; Gordon Cheung; Janet Dukerich; Jeff Edwards; Daniel Ganster; Mark Gavin; Jodi Goodman; Dave Harrison; Mickey Kavanagh; Adam Meade; Dan Newman; Catherine Schwoerer; Byran Schaffer; Tom Taber; Jeffrey Vancouver; James Vanscotter; Susan Winter

1. 1 IC = In Class Exam; TH = Take Home Exam