Psych 290: Basics of Design

Psych 290 |

Graduate Research Methods:
How to do stuff

back to course homepage

O. Outline

How do we come up with ideas?
Some basics
Quasi-experimental designs
Counterbalancing
Getting the most/best data
Methodology is subjective
Keep it simple, make it interesting
When might you not use a 7-point scale?

I. How do we come up with ideas?

read things you don’t agree with

if so, what’s the first thing to do? replicate, then change only one thing at a time

read things you agree with and want to test further

if so, what do you do? see what variables something varies with, see what other contexts it will work in - see if it’ll work in some realworld setting

observe a phenomenon in the world that you find interesting

the point of science is to come up with an explanation:

try reduce to some simple replicable experimental task

read a bit outside the field and find a theory that hasn’t been tested
try to find connections between bits of literature - then show how they’re related
tell your non-specialist friends about your research: most of my ideas come because of conversations with other people - because of the questions they ask, etc

II. Some basics:

external validity: random sampling
internal validity: random assignment to conditions
what do you need to have a proper experiment? (2 is more important)
independent variables (the things you manipulate)
dependent variables (the things you measure)
e.g., I make an experiment where letters of different sizes and fonts will appear on the screen and ask subjects to call them out as quickly as they can recognize the letter - I measure reaction time. what are the dependent & independent variables?
e.g., I wonder if men and women perform similarly under different test circumstances. I get a group of men and women to take an intelligence test in a quiet room or in a noisy room & measure their score. what are the dependent & independent variables?
aha! are we really doing random assignment with gender?
no, so it’s not an experiment - it’s a quasi-experimental design!

III. Quasi-experimental designs

psychologists use these all the time:

social: race, gender, different cultures

cognitive: different language speakers or high-low performers

personality: depressed vs. normal

neuro: neurological patient vs. normal

devo: different age groups

why are these quasi-experimental designs?
because we can’t randomly assign someone to be male or female, or to be black or white - this means we don’t have internal validity to the experiment.
what problems does this cause?
you can’t tell which of the many differences between men and women are causing the effect
you have to worry about fancy interactions with your design:
for example, suppose you’re conducting quality-of-life interviews with old and young people if both groups are interviewed by a young person, and there are differences, how do you know that it’s not because the old group just acts different in front of younger people - maybe if they were interviewed by someone their age, there would be no difference?
how to overcome this?
1. counterbalance: 1/2 old, 1/2 young experimenter for both old and young participants
2. better yet, show an interaction, not just a main effect difference: (or a double-dissociation, if you’re lucky)
it’s a platitude to say that men are different from women or that old people are different from young people, or that brain damage makes you worse at thinking - we all know that
it gets interesting when you can make really specific predictions: when you can show a specific way in which they’re different
so, what you need is an interaction - something where they’re different, and something where they’re not (or the difference is reversed). e.g., old people are worse at remembering the detials of a story than young people - but better at remembering the gist of a story. men are better at spatial tasks, but women are better at verbal tasks. a patient is impaired in long-term memory, but fine in short-term memory

IV. Counterbalancing:

say you want to give people 4 different personality surveys - or show them 4 different film clips & have them rate how funny they are.... should you just put them in some random order, the same for everyone?
no - you must counterbalance the order:
how many different orders are possible? 4! = 4*3*2*1 = 24
that’s a pain if you also have other manipulations, like male/female or bright-room/dark-room etc, etc. all of a sudden your complete counterbalance could be 24*2*2=96 subjects!
that sucks - how do you get around it?
use the latin square counterbalancing scheme - it’s not perfect, but it’ll do for most practical purposes.
Suppose we call the 4 film-clips A, B, C, and D. Each column below represents one ordering (column 1 for example is if you were to present your film-clips in the order ABCD). First we want to make sure that each film-clip gets presented first, second, third, and last an equal number of times. The easiest way to do this is to just rotate the ordering in column 1 all the way around (this way each clip gets to be first, second, third and last exactly once). That's how columns 2-4 were created.

	Col 1	Col 2	Col 3	Col 4
Row1	A	B	C	D
Row 2	B	C	D	A
Row 3	C	D	A	B
Row 4	D	A	B	C

But we're not done yet. We still have the problem that clips tend to follow the same clip in the ordering (for example, C follows B 3 out 4 times) - we want to mix that up a bit.
So, we obtain a random order for columns (draw column numbers without replacement by lottery).
Suppose the random order you drew is 2, 1, 4 and 3. We just move the columns around to get this layout:

	Col 1	Col 2	Col 3	Col 4
Row1	B	A	D	C
Row 2	C	B	A	D
Row 3	D	C	B	A
Row 4	A	D	C	B

Now, obtain a random order for the rows (using the same method as for columns).
If the order is 4, 2, 3 and 1, the final layout is:

	Col 1	Col 2	Col 3	Col 4
Row1	A	D	C	B
Row 2	C	B	A	D
Row 3	D	C	B	A
Row 4	B	A	D	C

How do you know when to randomize and when to counterbalance? if you have lots of subjects or lots of items, just randomize - if you have few subjects (e.g. baby studies, fMRI studies, you must counterbalance)
say you want to find out whether people like or dislike size-change. you show them a bunch of pictures, and then show them some unchanged pictures, or some pictures that are slightly larger, and get their liking ratings. is this a good design?

V. Getting the most/best data: N’s, designs, and replications

Between-subject vs. Within-subject designs
what are the benefits of each?
what are the compromises of each?
Maximizing data intake:
turn a between-subjects design into a within-subjects design - you can always throw away the second task subjects do if you think it's contaminated by the first task
come up with more measures - rather than just getting a subject’s forced-choice answer, add a confidence measure - if your manipulation is only having a small effect on a subject's decision, you’ll be able to see it in the confidence measure better than the multiple choice measure
shorten your task so that you can give it out to people at the mall or at the airport - there is nothing better than getting 300 subjects in an afternoon
piloting: always run yourself through the experiment first, and run a few of the subjects yourself (if your design allows it) - you want to see what the subjects are doing - how they’re responding - people are incredibly tricky things
it doesn’t hurt to ask your subjects what they thought of the task - what strategies they used etc - they’ll tell you a lot of useful stuff.
replications:
suppose you’ve just done a difficult and time-consuming study with 40 subjects. you’ve scored and analyzed a large number of variables. most results are inconclusive, but one before/after comparison is highly significant t=2.70, which is surprising and could be of major theoretical importance.
1. should you replicate the study before publishing?
2. if yes, how many subjects should you replicate with?
answer: more than you had in the original set.
BUT: why are you replicating? you should be pooling the data!
saves you frustration...
read the Tversky paper that I gave out in class for all the details

VI. Methodology is subjective

There is no one best way to run any study or to test any hypothesis - there are always many ways. What is and isn’t necessary to control for will get decided as part of scientific dialogue. So much of what is considered good methodology is subjective and mostly depends on whether what you do will convince or not convince your audience.
The areas that are most highly controversial, are the ones with the best methodological standards - they have to use the highest precautions because otherwise noone will believe them. One of the most controversial areas is Psi phenomena (ESP, clairvoyance, telekenesis, etc). Oddly enough, this is also a good area to learn about methodology (both from past mistakes, and from current precautions). The Bem paper I gave out in class is an excellent place to learn how to be convincing - no matter how crazy the thing you're testing is.
Suppose we want to find out whether certain phenomenna exist - how do we test - let’s come up with as many ways as possible....

VII. Keep it simple, make it interesting:

when designing a study, ask yourself two things:

1. how am I going to analyze the data?

2. if the New York Times ran a story on this study, what would the headline be?
asking yourself these two things will give your designs more clarity. this is good both for the internal validity of your experiment, and also for other people's ability to understand it and tell others about it.
keep it simple! the simpler your stimuli and manipulations, the fewer possibilities for potential confounds
no experiment is perfect - there will always be some problems, some confounds: always try to show the same effect with different stimuli and in different paradigms
always do at least one "cute" or "cocktail party" study - this is the study people will talk about and this is how they’ll get to know your work. the study needs to be easy to remember and easy to explain. it need not be the cleanest or the most sophisticated study - just the most memorable one.
let’s take an example: change-blindness has been around for a long time. one could do a study where you change N number of pixels in a picture and get RT’s for how long it takes people to notice where they are, and then you can make a very scientific-looking graph. or you could make a video of the "door" experiment - and the everyone will talk about it and get all excited.
find a good phenomenon - something you can show people that’s not just a bar-graph. it’s ok to say 50% of people don’t notice that it’s a different guy asking directions, but what’s really convincing is when you see the video - people love to see a phenomenon with their own eyes. transcripts & other phenomenon-capturing data types work well too.

VIII. Making Scales

When might you not use a 7-point scale?

recent NY Times article on scale construction
(you can get a FREE online subscription to the NY Times just by registering)