Cognitive Psychology Series Part 13: Research Methods

1. Experimental Design

Experimental design is the blueprint of a study -- the strategic plan for how participants will be assigned to conditions, what variables will be manipulated and measured, and how confounds will be controlled. The choice of design has profound implications for what conclusions can be drawn.

1.1 Between-Subjects Design

In a between-subjects (independent groups) design, each participant experiences only one level of the independent variable. Different groups of participants are compared.

Example: To test whether background music affects reading comprehension, Group A reads in silence, Group B reads with classical music, and Group C reads with pop music. Each participant is in only one condition.

Key control technique: Random assignment -- participants are randomly allocated to conditions, ensuring that pre-existing differences (age, IQ, motivation) are distributed equally across groups. Without random assignment, you have a quasi-experiment, not a true experiment.

1.2 Within-Subjects (Repeated Measures) Design

In a within-subjects design, every participant experiences all levels of the independent variable. The same person serves as their own control.

Example: Each participant completes a Stroop task under three conditions: congruent (word "RED" in red ink), incongruent (word "RED" in blue ink), and neutral (a string of Xs in colored ink). Their reaction times are compared across all three conditions.

Key control technique: Counterbalancing -- varying the order of conditions across participants (e.g., half do condition A then B; the other half do B then A) to cancel out order effects. A full Latin Square design systematically rotates all possible orderings.

1.3 Factorial & Mixed Designs

A factorial design examines two or more independent variables simultaneously, allowing researchers to detect interactions -- situations where the effect of one variable depends on the level of another.

Example of a 2 x 3 factorial design: Factor A = Encoding type (visual vs verbal), Factor B = Retention interval (1 hour, 1 day, 1 week). This yields 6 conditions and can reveal whether the decay rate differs for visual vs verbal memories.

A mixed design combines between-subjects and within-subjects factors. For instance, comparing older adults vs younger adults (between-subjects) on a memory task measured at multiple time points (within-subjects).

1.4 Quasi-Experimental Design

When true random assignment is impossible -- because the independent variable is a pre-existing characteristic (age, clinical diagnosis, handedness) or an event that cannot be ethically manipulated -- researchers use quasi-experimental designs.

Example: Comparing cognitive function in patients with Alzheimer's disease vs healthy controls. You cannot randomly assign people to have Alzheimer's, so groups differ in ways beyond just the variable of interest.

2. Variables & Hypothesis Testing

2.1 Independent, Dependent & Confounding Variables

Every experiment revolves around three types of variables:

Extraneous variables are any variables other than the IV that could affect the DV. They become confounds only when they systematically co-vary with the IV. Good experimental design uses randomization, counterbalancing, and standardization to prevent extraneous variables from becoming confounds.

2.2 Hypothesis Testing: Null vs Alternative

The logic of null hypothesis significance testing (NHST) -- the dominant statistical framework in psychology -- works by assuming the null hypothesis (H0: there is no effect) is true, then calculating the probability of obtaining data as extreme as what was observed.

2.3 Effect Size & Power Analysis

Statistical significance tells you whether an effect is likely real, but not whether it matters. That is the role of effect size -- a standardized measure of the magnitude of an effect.

Statistical power is the probability of correctly detecting a real effect (1 - beta). A well-powered study typically aims for 80% power. Power depends on three factors: effect size (larger = easier to detect), sample size (larger = more power), and alpha level (more lenient = more power but more Type I errors).

# Power analysis and effect size calculation in Python
import math
import random

def cohens_d(group1, group2):
    """Calculate Cohen's d for two independent groups."""
    n1, n2 = len(group1), len(group2)
    mean1, mean2 = sum(group1) / n1, sum(group2) / n2

    # Pooled standard deviation
    var1 = sum((x - mean1) ** 2 for x in group1) / (n1 - 1)
    var2 = sum((x - mean2) ** 2 for x in group2) / (n2 - 1)
    pooled_sd = math.sqrt(((n1 - 1) * var1 + (n2 - 1) * var2) / (n1 + n2 - 2))

    d = (mean1 - mean2) / pooled_sd
    return d

def required_sample_size(effect_size, alpha=0.05, power=0.80):
    """
    Approximate sample size per group for a two-sample t-test.
    Uses the formula: n = (z_alpha + z_beta)^2 * 2 / d^2
    """
    # Z-scores for common alpha and power levels
    z_alpha = 1.96 if alpha == 0.05 else 2.576  # two-tailed
    z_beta = 0.84 if power == 0.80 else 1.28    # power = .80 or .90

    n_per_group = math.ceil((z_alpha + z_beta) ** 2 * 2 / effect_size ** 2)
    return n_per_group

# Simulate a Stroop experiment
random.seed(42)
congruent_rts = [random.gauss(520, 80) for _ in range(50)]    # ms
incongruent_rts = [random.gauss(620, 95) for _ in range(50)]  # ms

d = cohens_d(incongruent_rts, congruent_rts)
print(f"=== Stroop Effect Simulation ===")
print(f"Congruent mean RT: {sum(congruent_rts)/len(congruent_rts):.1f} ms")
print(f"Incongruent mean RT: {sum(incongruent_rts)/len(incongruent_rts):.1f} ms")
print(f"Cohen's d: {abs(d):.3f} (large effect)")

# Power analysis for different effect sizes
print(f"\n=== Required Sample Sizes (alpha=.05, power=.80) ===")
for label, es in [("Small (d=0.2)", 0.2), ("Medium (d=0.5)", 0.5), ("Large (d=0.8)", 0.8)]:
    n = required_sample_size(es)
    print(f"  {label}: {n} participants per group ({n*2} total)")

3. Statistical Analysis

Choosing the correct statistical test depends on the research design, the type of data, and the research question. Here is a practical guide to the most commonly used tests in cognitive psychology research.

3.1 t-Tests

The t-test compares means from two conditions to determine whether they differ significantly.

3.2 Analysis of Variance (ANOVA)

When you have three or more conditions, multiple t-tests inflate your Type I error rate. ANOVA solves this by testing whether any group means differ significantly in a single omnibus test.

# Simulating a One-Way ANOVA: Effect of background music on memory recall
import random
import math

def one_way_anova(groups):
    """
    Perform a one-way ANOVA from scratch.
    Returns F-statistic and effect size (eta-squared).
    """
    k = len(groups)
    all_data = [x for g in groups for x in g]
    N = len(all_data)
    grand_mean = sum(all_data) / N

    # Sum of Squares Between (SSB)
    ssb = sum(len(g) * (sum(g)/len(g) - grand_mean)**2 for g in groups)

    # Sum of Squares Within (SSW)
    ssw = sum(sum((x - sum(g)/len(g))**2 for x in g) for g in groups)

    # Degrees of freedom
    df_between = k - 1
    df_within = N - k

    # Mean squares
    msb = ssb / df_between
    msw = ssw / df_within

    # F-statistic
    f_stat = msb / msw

    # Effect size (eta-squared)
    eta_sq = ssb / (ssb + ssw)

    return f_stat, df_between, df_within, eta_sq

# Simulate three conditions
random.seed(42)
silence = [random.gauss(15, 3) for _ in range(30)]      # 15 words recalled
classical = [random.gauss(14, 3) for _ in range(30)]     # 14 words recalled
pop_music = [random.gauss(11, 3.5) for _ in range(30)]   # 11 words recalled

f, df1, df2, eta = one_way_anova([silence, classical, pop_music])

print("=== One-Way ANOVA: Background Music & Memory ===")
print(f"Silence:   M = {sum(silence)/len(silence):.1f} words")
print(f"Classical: M = {sum(classical)/len(classical):.1f} words")
print(f"Pop Music: M = {sum(pop_music)/len(pop_music):.1f} words")
print(f"\nF({df1}, {df2}) = {f:.2f}")
print(f"eta-squared = {eta:.3f} ({'large' if eta > .14 else 'medium' if eta > .06 else 'small'} effect)")
print(f"{'Significant at p < .05' if f > 3.10 else 'Not significant'}")

3.3 Correlation & Regression

Correlation measures the strength and direction of the linear relationship between two variables. Regression goes further by modeling one variable as a function of one or more predictors.

3.4 Choosing the Right Statistical Test

4. Cognitive Experimental Paradigms

Cognitive psychologists have developed a remarkable toolkit of experimental paradigms -- standardized tasks that reliably tap specific cognitive processes. These paradigms are the workhorses of the field, used in thousands of studies across labs worldwide.

4.1 The Stroop Task

The Stroop task (Stroop, 1935) is arguably the most famous paradigm in cognitive psychology. Participants must name the ink color of printed words while ignoring the word itself. When the word and ink color conflict (e.g., the word "RED" printed in blue ink), reaction times increase dramatically -- the Stroop effect.

4.2 The Eriksen Flanker Task

The flanker task (Eriksen & Eriksen, 1974) measures the ability to suppress responses to irrelevant stimuli surrounding a target. Participants respond to a central stimulus (e.g., the direction of a central arrow) while ignoring flanking distractors.

Example stimuli:

• Congruent: > > > > > (all arrows point right) -- Fast, accurate
• Incongruent: < < > < < (flankers conflict with target) -- Slower, more errors
• Neutral: -- -- > -- -- (non-arrow flankers) -- Intermediate

The flanker effect demonstrates that selective attention has spatial limits -- nearby distractors are processed even when they are irrelevant, particularly when they are close to the target. This paradigm is central to theories of response competition and attentional filtering.

4.3 Go/No-Go, N-Back & Visual Search

5. Reaction Time Studies

Reaction time (RT) is the primary dependent variable in cognitive psychology -- a millisecond-precise window into the speed of mental processing. The logic is simple but powerful: if manipulating a variable increases RT, that manipulation has added cognitive processing demands.

5.1 Donders' Subtraction Method

Franciscus Donders (1868) developed the subtraction method to estimate the duration of specific mental processes. He designed three types of reaction time tasks, each adding one additional cognitive operation:

By subtracting task durations, Donders estimated: Discrimination time = C - A = ~85 ms; Response selection time = B - C = ~85 ms. This elegant logic assumes that cognitive processes are additive and independent -- an assumption later challenged by Sternberg's additive factors method (1969).

5.2 Speed-Accuracy Tradeoff

One of the most fundamental constraints in human information processing is the speed-accuracy tradeoff (SAT): faster responses tend to be less accurate, and more accurate responses tend to be slower. Participants can shift their criterion along this continuum.

5.3 Hick's Law

Hick's Law (Hick, 1952; Hyman, 1953) states that choice reaction time increases logarithmically with the number of response alternatives:

RT = a + b * log2(n)

where n is the number of equally probable alternatives, a is the base RT, and b is the slope (about 150 ms per bit of information). This relationship shows that the human decision-making system processes information in bits, much like a digital system, supporting the information-processing metaphor central to cognitive psychology.

Practical application: Hick's Law directly influences UX design. Menus with fewer options lead to faster selection times. This is why simplified navigation (e.g., 5-7 main menu items) leads to better user experience than presenting 20+ options simultaneously.

6. Neuroimaging Methods in Cognitive Research

Modern cognitive psychology increasingly integrates neuroimaging -- techniques that measure brain activity during cognitive tasks. Each method offers a different tradeoff between spatial resolution (where in the brain) and temporal resolution (when activity occurs).

7. Replication Crisis & Open Science

7.1 Ecological Validity

Ecological validity refers to the degree to which experimental findings generalize to real-world settings. A perennial tension in cognitive psychology is between internal validity (controlled lab conditions) and ecological validity (real-world relevance).

Consider memory research: studying word list recall in a quiet lab has high internal validity but may tell us little about how memory operates when navigating a busy city, having a conversation, or studying for an exam while distracted by social media. Neisser (1976) famously criticized the field for studying memory "in a vacuum," calling for more ecologically valid research paradigms.

Modern responses to this challenge include experience sampling methods (ESM), virtual reality experiments, and large-scale online studies that sacrifice some control for greater ecological representativeness.

7.2 The Replication Crisis

In 2015, the Open Science Collaboration attempted to replicate 100 published psychology experiments. The results were sobering: while 97% of the original studies reported significant results, only 36% of replications yielded significant effects. Effect sizes in replications were, on average, half the magnitude of the originals.

Several factors contributed to the crisis:

• Publication bias: Journals overwhelmingly publish positive results, creating a "file drawer problem" where null results are never shared
• p-hacking: Researchers (often unconsciously) make analysis decisions that inflate p-values -- testing multiple DVs, removing outliers, adding covariates, or stopping data collection when p < .05
• HARKing: Hypothesizing After Results are Known -- presenting post-hoc findings as if they were predicted a priori
• Underpowered studies: Many studies had too few participants to reliably detect the effects they claimed to find

7.3 The Open Science Movement

The replication crisis catalyzed a powerful reform movement. The open science movement promotes transparency, rigor, and reproducibility through concrete practices:

7.4 Meta-Analysis

Meta-analysis is a statistical technique that synthesizes findings from multiple studies on the same topic, providing a more precise estimate of the true effect size than any single study can.

Steps in conducting a meta-analysis:

1. Define the research question and inclusion criteria
2. Systematic literature search -- exhaustive, documented search of databases
3. Code studies -- extract effect sizes, sample sizes, and moderator variables
4. Compute weighted average effect size -- larger studies receive more weight
5. Test for heterogeneity -- are effect sizes consistent across studies?
6. Analyze moderators -- what factors explain variation in effect sizes?
7. Assess publication bias -- funnel plots and trim-and-fill analysis

Exercises & Self-Assessment

Conclusion & Next Steps

In this penultimate chapter of our Cognitive Psychology Series, we have examined the scientific methods that underpin everything cognitive psychologists claim to know about the mind. Here are the key takeaways:

• Experimental design is the foundation of causal inference. Between-subjects, within-subjects, factorial, and quasi-experimental designs each have distinct strengths and limitations. Random assignment is essential for causation.
• Hypothesis testing is widely used but widely misunderstood. A p-value is not the probability that H0 is true. Effect size and power are at least as important as significance.
• Statistical tests should match the research design: t-tests for two conditions, ANOVA for three or more, correlation and regression for relationships, chi-square for categorical data.
• Cognitive paradigms like the Stroop, flanker, and n-back tasks are the workhorses of cognitive research, providing reliable windows into specific processes like attention, inhibition, and working memory.
• Reaction time is the gold standard DV in cognitive psychology, dating back to Donders' subtraction method. The speed-accuracy tradeoff and Hick's law reveal fundamental constraints of the information-processing system.
• Neuroimaging methods complement behavioral measures, each with different spatial and temporal resolution tradeoffs. Converging evidence across methods yields the strongest conclusions.
• The replication crisis exposed real problems in research practice, but the open science response -- pre-registration, open data, registered reports -- is making the field more rigorous and trustworthy.