Research Designs

A. Overview of the problem

1. Theoretical problem: understanding the relationship between latent variables (constructs)

a) relationships among latent variables
b) relationships between latent variables and observed variables

2. Generalization of results and threats to external validity

a) limitations of generalization for subjects
- (1) freshman psych students at NU
- (2) students at NU
- (3) college students at selective research universities
- (4) college students
- (5) 18-24 year olds
- (6) North Americans
b) limits of generalization for conditions -- interactions with other variables
- (1) problems and benefits of interactions
  - (a) xy relationship depends upon z
  - (b) example:
    - i) in the morning, caffeine hinders working memory performance
    - ii) in the evening, caffeine facilitates working memory performance
- (2) interactions limit generalization
- (3) interactions test theoretical limits

B. Practical problems and threats to internal validity

1. manipulations affect more than the construct of interest

a) examples:
- (1) caffeine induces alertness and motor tremor
- (2) failure induces anxiety, depression, anger
- (3) practice leads to motivational changes as well as changes in skill

2. observable variables reflect more than the construct of interest

a) examples: self report of alertness reflects base line differences
b) cognitive performance--ability, motivation, training, practice

C. Types of designs

1. Between subject designs

a) Subject variables as an alternative explanation of results -- threats to validity
- (1) Examples of sources of subject variables
  - (a) ability
  - (b) practice
  - (c) motivation
  - (d) interest
  - (e) gender
  - (f) age
- (2) unintended variations that can lead to subject variability
  - (a) time of day
    - i) naturally occuring rhythm of alertness
    - ii) classroom effects
    - iii) fatigue
  - (b) time of week, month, season, year
    - i) class schedules
      - (1) mid terms
      - (2) papers
    - ii) weather
  - (c) volunteer effects
  - (d) experimenter - subject interaction effects
b) Randomization as a control
- (1) Only the expected values of groups are equal -- not the observed values
  - (a) any particular experiment, groups are not equivalent
  - (b) expected value of the (signed) group difference = 0
- (2) randomization does not introduce systematic bias
- (3) types of randomization
  - (a) subjects matched on variable of interest and then assigned to conditions
  - (b) blocking as a control for order effects
    - i) ignores stable subject effects
    - ii) eliminates subject effects associated with time of appearance
  - (c) complete randomization
    - i) ³failures² of randomization
    - ii) problem at the end of the experiment
      - (1) power is maximal if sample sizes are equal
      - (2) randomization will tend not to produce equal size groups

2. Within subject designs

a) controls for subject variability
- (1) every subject is his/her own control
- (2) particularly important if there are large between subject effects
  - (a) reaction time
    - i) simple
    - ii) complex or choice rt
      - (1) lexical decision
      - (2) color naming
  - (b) cognitive performance
b) confounds practice/order effects with manipulation
- (1) practice/order effects
  - (a) improvement in skill
  - (b) decrease in interest
  - (c) fatigue
- (2) differential transfer
  - (a) prior experience changes state
    - i) exposure to drug
    - ii) exposure to memory recall/recognition
c) two or more conditions -- repeated many, many times
- (1) block randomization within subject
  - (a) ABCCBAACB ...... BCA
- (2) effect of any single order washes out
d) conditions- repeated twice
- (1) ABBA or BAAB counterbalancing
- (2) ABCCBA or BCAACB or CABBAC counterbalancing
- (3) need to have subjects in different orders to control for possible order effects

3. Mixed -- Within/Between

a) Between as the order variable -- not theoretically interesting
- (1) conditions- repeated twice
  - (a) ABBA or BAAB counterbalancing
  - (b) ABCCBA or BCAACB or CABBAC counterbalancing
  - (c) need to have subjects in different orders to control for possible order effects
- (2) multiple conditions -- multiple but limited trials
  - (a) Latin square design can control across subjects for order and some sequence effects
  - (b) sample Latin Square
    - i) ABCD
    - ii) BDAC
    - iii) CADB
    - iv) DCBA
b) Between subjects as a manipulation of an IV -- theoretically interesting
- (1) Example: effect of caffeine and time of day on complex reasoning
  - (a) Between subjects
    - i) caffeine/placebo
    - ii) morning/evening
  - (b) within subjects
    - i) easy/hard problems
    - ii) rt over trials

D. Designs- the problems and benefits of subject variables

1. Subject variables as necessary in psychological studies

a) if people do not differ on a variable, is it a psychological variable?
b) subject variables are either part of theory or extraneous to theory

2. Subject variables as sources of variance

a) noise variance (unsystematic)
b) confounded variance (systematic

3. Controlling unsystematic Subject Variance

a) within subject design controls for ability and motivational differences
- (1) trait variables
  - (a) ability
  - (b) age
  - (c) prior practice
- (2) state variables
  - (a) prior practice
  - (b) interest
b) within subject designs lead to problems of order, fatigue, practice
- (1) controlled by counterbalancing
- (2) does not protect against differential transfer

4. Subject variables as systematic sources of variance

a) subject attrition
b) subject volunteer effects
c) subject by task interactions
d) subjects by manipulation interactions

Part of a course in Research Methods in Psychology (Psychology 205) at Northwestern University, developed by William Revelle.
Back to the course syllabus.
Revised April 16, 2002