This guide examines reliability vs validity psychology, test retest vs inter rater reliability, construct vs criterion validity psychology, and why a test can be reliable but not valid alongside our research methods resources and our psychology variables and operational definitions worksheet.
OpenStax Psychology 2e (published Apr 22, 2020 by Rose M. Spielman, William J. Jenkins, and Marilyn D. Lovett) includes Section 2.3 Analyzing Findings.
An operational definition is a precise description of variables, allowing others to understand exactly how and what a researcher measures in a particular experiment.
A correlation coefficient r is a number from -1 to +1 that indicates the strength and direction of the relationship between variables.
A correlation coefficient of 0.9 indicates a far stronger relationship than a correlation coefficient of 0.3, and if variables are not related at all, the correlation coefficient is 0.
| Measurement Property | Psychometric Category | Definition in Source Literature |
|---|---|---|
| Overall consistency | Reliability | Reliability refers to the ability to consistently produce a given result, meaning that any instruments or tools used to collect data in psychological research do so in consistent, reproducible ways. |
| Observer agreement | Inter-rater reliability | Inter-rater reliability is the degree to which two or more different observers agree on what has been observed. |
| Item and temporal consistency | Internal consistency and test-retest reliability | Internal consistency is the degree to which different items on a survey that measure the same thing correlate with one another, and test-retest reliability is the degree to which outcomes of a measure remain consistent over multiple administrations. |
| Overall accuracy | Validity | Validity refers to the extent to which a given instrument or tool accurately measures what it is supposed to measure. |
| Surface, construct, and real-world scope | Ecological, construct, and face validity | Ecological validity is the degree to which research results generalize to real-world applications, construct validity is the degree to which a variable actually captures or measures what it is intended to measure, and face validity is the degree to which a variable seems valid on the surface. |
| Criterion timing | Concurrent and predictive validity | When the criterion is measured at the same time as the construct, criterion validity is referred to as concurrent validity; when the criterion is measured at some point in the future after the construct, it is referred to as predictive validity. |
Core Definitions: True Variance, Error Variance, and Score Inferences
Research Methods in Psychology (Copyright 2017 by Paul C. Price, Rajiv Jhangiani, I-Chant Chiang, Dana C. Leighton, and Carrie Cuttler) includes Section 4.2 Reliability and Validity of Measurement.
In the REL Midwest Reliability and Validity Handout, reliability is the relationship among items within an assessment or between different administrations, and is an estimate of how closely an observed score is related to what the true score would be if measured perfectly.
In Koo and Li (issue date 2016 Jun), reliability represents a ratio of true variance over true variance plus error variance (ranging between 0 and 1); in Table 1, true variance 9.6° and error variance 12.8° (measured variance 22.4°) yield a reliability index of 9.6 / (9.6 + 12.8) = 0.43.
In the REL Midwest Reliability and Validity Handout, validity is the degree to which evidence and theory support the use of a measure for a specific purpose, giving an estimate of the usefulness of inferences drawn from test scores.
Can a Test Be Reliable But Not Valid: The Asymmetry Rule and Systematic Bias
While any valid measure is by necessity reliable, the reverse is not necessarily true, and researchers strive to use instruments that are both highly reliable and valid.
In Tavakol and Dennick (accepted 2011 Jun 17), an instrument cannot be valid unless it is reliable, whereas the reliability of an instrument does not depend on its validity.
If a kitchen scale is not properly calibrated, it may consistently under- or overestimate the amount of cereal poured onto it; while highly reliable in producing consistent results, those results are incorrect.
Measuring index finger length with a ruler to assess self-esteem would have extremely good test-retest reliability but absolutely no validity, because one person’s index finger being a centimeter longer than another’s indicates nothing about self-esteem.
Photograph of an analog kitchen scale with a bowl of cereal beside a wooden strip and reading glasses on a study table.
Psychological Constructs, Operational Scoring, and Levels of Measurement
In Section 4.1 Understanding Psychological Measurement, the Beck Depression Inventory is a 21-item self-report questionnaire in which a person rates sadness, energy loss, and other depression symptoms over the past 2 weeks, and the sum of the 21 ratings represents current depression.
The Big Five is a set of five broad personality dimensions, and each of the Big Five can be defined in terms of six more specific constructs called facets (Costa & McCrae, 1992).
On the Social Readjustment Rating Scale illustration, a man who has been divorced (73 points), changed jobs (36 points), and had a change in sleeping habits (16 points) in the past year would have a total score of 125.
Because IQ is measured at the interval level rather than the ratio level, one cannot say that someone with an IQ of 140 is twice as intelligent as someone with an IQ of 70.
Test-Retest vs. Inter-Rater Reliability and Internal Consistency
For additional examples on reading reliability statistics in primary journal articles, see our guide on how to read a psychology study and our correlation and causation headline rewrite activity.
Test-Retest Reliability, Systematic Shift, and the Reliability Paradox
In Figure 4.2, the test-retest correlation between two sets of scores of university students on the Rosenberg Self-Esteem Scale administered two times a week apart is +.95, and in general a test-retest correlation of +.80 or greater is considered to indicate good reliability.
Because the very nature of mood is that it changes, a measure of mood that produced a low test-retest correlation over a period of a month would not be a cause for concern.
Pearson’s correlation r is inappropriate for test-retest reliability because if five individuals score 30, 35, 40, 45, and 50 at Time 1 and then score 40, 45, 50, 55, and 60 two weeks later, Pearson’s r will be 1 (perfect correlation) even though the scores changed substantially.
McGraw and Wong (1996) defined 10 forms of ICC and Shrout and Fleiss (1979) defined 6 forms of ICC (eg, ICC (2,1)), and the 2-way mixed-effects model with absolute agreement definition should be used for test-retest and intrarater reliability studies.
As a rule of thumb, researchers should obtain at least 30 heterogeneous samples and involve at least 3 raters for a reliability study, and based on the 95% confidence interval of the ICC estimate, values less than 0.5 indicate poor, 0.5 to 0.75 moderate, 0.75 to 0.9 good, and greater than 0.90 excellent reliability.
In Hedge, Powell, and Sumner (issue date 2018), three-week test-retest reliabilities (calculated with ICC (2,1), two-way random effects for absolute agreement) across seven classic cognitive tasks (Eriksen Flanker, Stroop, stop-signal, go/no-go, Posner cueing, Navon, and SNARC) ranged from 0 to .82 because low between-subject variance causes low reliability for individual differences.
In Hedge, Powell, and Sumner Studies 1 and 2, none of the behavioral measures exceeded reliabilities of .8; the Flanker RT cost had ICCs of .40 and .57 (with SEM = 15 ms in both), the Stroop RT cost had ICCs of .60 and .66, go/no-go commission errors had ICC = .76 in both studies, and UPPS-P Sensation Seeking had ICCs of .87 and .89.
Inter-Rater Reliability: Behavioral Coding, Observer Agreement, and Cohen’s Kappa
In Bandura, Ross, and Ross (1961), children played for 20 minutes in a room containing a Bobo doll, and physical aggression was operationally defined as the number of specifically defined aggressive acts committed during the 20-minute period.
Inter-rater reliability is the degree to which two or more different observers agree on what has been observed.
Interrater reliability is often assessed using Cronbach’s α when judgments are quantitative or an analogous statistic called Cohen’s κ (the Greek letter kappa) when judgments are categorical.
An inter-rater reliability (IRR) check needs to be carried out separately for every tool used in a study with multiple outcome raters, even if a rater scored only 10% of the study sample.
In McHugh (collection date 2012 Oct), Jacob Cohen (1960) critiqued percent agreement for its inability to account for chance agreement and introduced Cohen’s kappa (κ), which ranges from −1 to +1, where 0 represents random chance agreement and 1 represents perfect agreement.
In McHugh (2012) Table 1, two raters (Mark and Susan) rating 10 binary variables have 8 zeros (agreements) out of 10 items, yielding 80% percent agreement and meaning that 20% of the data collected in the study is erroneous.
Cohen suggested interpreting kappa values ≤ 0 as no agreement, 0.01–0.20 as none to slight, 0.21–0.40 as fair, 0.41–0.60 as moderate, 0.61–0.80 as substantial, and 0.81–1.00 as almost perfect agreement, whereas McHugh states that any kappa below 0.60 indicates inadequate agreement.
In Table 3 (Interpretation of Cohen’s kappa), the value of kappa, level of agreement, and percent of data that are reliable are: 0–.20 None (0–4%), .21–.39 Minimal (4–15%), .40–.59 Weak (15–35%), .60–.79 Moderate (35–63%), .80–.90 Strong (64–81%), and Above .90 Almost Perfect (82–100%).
A study comparing human visual detection with automated detection of abnormalities in biological samples found an excellent percent agreement of 94.2% but only a moderate kappa (κ = 0.555), and as a general heuristic sample sizes for kappa should not consist of less than 30 comparisons.
Photograph of parallel clipboards, pens, and stopwatches arranged in front of a behavioral observation window.
Internal Consistency: Split-Half Reliability, Cronbach’s Alpha, and Error Variance
In Figure 4.3, the split-half correlation between university students’ scores on the even-numbered items and odd-numbered items of the Rosenberg Self-Esteem Scale is +.88, and a split-half correlation of +.80 or greater is generally considered good internal consistency.
There are 252 ways to split a set of 10 items into two sets of five; conceptually, Cronbach’s α is the mean of all 252 split-half correlations, and a value of +.80 or greater is generally taken to indicate good internal consistency.
Alpha was developed by Lee Cronbach in 1951 to provide a measure of the internal consistency of a test or scale, and it is expressed as a number between 0 and 1.
Squaring a test’s reliability correlation and subtracting from 1.00 produces the index of measurement error: if a test has a reliability of 0.80, there is 0.36 error variance (random error) in the scores (0.80×0.80 = 0.64; 1.00 – 0.64 = 0.36).
Cronbach’s alpha is grounded in the tau-equivalent model (assuming each test item measures the same latent trait on the same scale) and is a lower-bound estimate of reliability, so violating tau-equivalence with multiple factors or too small a number of items underestimates reliability.
There are different reports about the acceptable values of alpha, ranging from 0.70 to 0.95, and a maximum alpha value of 0.90 has been recommended because a higher value may suggest that some items are redundant.
In Menon et al. (issue date 2025 Jul), conventional thresholds for interpreting Cronbach’s alpha are below .50 unacceptable, .51 to .60 poor, .61 to .70 questionable, .71 to .80 acceptable, .81 to .90 good, and .91 to .95 excellent, while above .95 is flagged for excessive overlap and possible redundancy.
A small tool (e.g., fewer than 5 items) may yield a falsely low value of Cronbach’s alpha, whereas alpha may be falsely high for a large tool.
A split-half reliability test is preferred over Cronbach’s alpha when there are many items in a tool (e.g., 100 or more), and for tools with dichotomous item data, Kuder and Richardson (1937) proposed KR20.
| Reliability Context | Statistical Estimator | Published Interpretation Benchmark |
|---|---|---|
| Temporal score consistency | Test-retest correlation | In Figure 4.2, the test-retest correlation between two sets of scores of university students on the Rosenberg Self-Esteem Scale administered two times a week apart is +.95, and in general a test-retest correlation of +.80 or greater is considered to indicate good reliability. |
| Quantitative agreement and sample design | Intraclass correlation coefficient | As a rule of thumb, researchers should obtain at least 30 heterogeneous samples and involve at least 3 raters for a reliability study, and based on the 95% confidence interval of the ICC estimate, values less than 0.5 indicate poor, 0.5 to 0.75 moderate, 0.75 to 0.9 good, and greater than 0.90 excellent reliability. |
| Categorical observer agreement | Cohen’s kappa statistic | Cohen suggested interpreting kappa values ≤ 0 as no agreement, 0.01–0.20 as none to slight, 0.21–0.40 as fair, 0.41–0.60 as moderate, 0.61–0.80 as substantial, and 0.81–1.00 as almost perfect agreement, whereas McHugh states that any kappa below 0.60 indicates inadequate agreement. |
| Multi-item survey consistency | Cronbach’s alpha | There are different reports about the acceptable values of alpha, ranging from 0.70 to 0.95, and a maximum alpha value of 0.90 has been recommended because a higher value may suggest that some items are redundant. |
Construct vs. Criterion Validity, Content Coverage, and Administration Threats
To practice evaluating study designs and measurement claims with self-check questions, visit our psychology research methods practice questions.
Face Validity and Content Validity Quantification
Ecological validity is the degree to which research results generalize to real-world applications, construct validity is the degree to which a variable actually captures or measures what it is intended to measure, and face validity is the degree to which a variable seems valid on the surface.
The Minnesota Multiphasic Personality Inventory-2 (MMPI-2) has people decide whether each of over 567 different statements applies to them and works well despite lacking face validity, where statements about detective stories or the sight of blood measure suppression of aggression.
Content validity can be quantified using assessments from expert judges (typically ranging from 5 to 7) via content validity ratio, content validity index, or Cohen’s coefficient kappa, and pre-testing ideally uses cognitive interviews of 5–15 interviews in two to three rounds (or until saturation).
Criterion Validity: Concurrent Versus Predictive Evidence
When the criterion is measured at the same time as the construct, criterion validity is referred to as concurrent validity; when the criterion is measured at some point in the future after the construct, it is referred to as predictive validity.
In Greca and Stone’s evaluation of concurrent validity, the 26-item revised social anxiety scale for children (SASC-R, adding 16 items to the 10-item SASC) showed Pearson product-moment validity coefficients of 0.94 for fear of negative evaluation (FNE) and 0.88 for social avoidance and distress (SAD and SAD-New).
College Board research suggests SAT scores have high predictive validity for first-year college students’ GPA (Kobrin et al., 2008), while Rothstein (2004) suggested that the SAT’s predictive validity may be overestimated by as much as 150%.
As of March 2019, more than 1000 institutions of higher education had either relaxed or eliminated the requirements for SAT or ACT testing for admissions (Strauss, 2019, March 19).
Construct Validity: Convergent, Discriminant, and Structural Threats
Four indicators of construct validity in scale development are convergent validity, discriminant validity, differentiation by known groups, and correlation analysis, where convergent and discriminant validity are best estimated through the multi-trait multi-method matrix (Campbell & Fiske, 1959).
Discriminant validity is the extent to which scores on a measure are not correlated with measures of conceptually distinct variables; if a new measure of self-esteem were highly correlated with mood, it could be argued that it is measuring mood instead.
In Boateng et al. (collection date 2018), two threats that can invalidate a scale are construct underrepresentation (when a scale does not capture important aspects of a construct because its focus is too narrow) and construct-irrelevant variance (when test scores are influenced by processes with little to do with the intended construct).
In Cromwell (18 December 2005), deaf school-leavers’ median reading age was estimated at the nine-year level irrespective of intellectual function (Conrad, 1979), and even for an NVQ Level IV accredited sign language interpreter, interpretation reliability declines significantly after 35 minutes (Brasel, 1976).
In Bartram (18 November 2005), other things being equal, test length is related to reliability and hence to validity, and the 2005 International Test Commission (ITC) computer-based testing guidelines distinguish four modes of internet test administration: Open mode, Controlled mode, Supervised mode, and Managed mode.
Photograph of a seminar desk arranged with blank index cards in a matrix layout beside stacked study folders.
Scale and Behavioral Observation Reliability–Validity Diagnostic Matrix
The reflection prompts below help students and instructors audit a psychological questionnaire or behavioral coding protocol.
| Diagnostic Area | Protocol Review Focus | Student Reflection Question |
|---|---|---|
| Construct stability | Matching the measurement interval to the theoretical nature of the variable | Is the psychological construct expected to remain stable across sessions or fluctuate like transient mood? |
| Agreement estimator | Choosing between linear association and absolute agreement across time points | Does the analysis check for systematic upward or downward score shifts across sessions rather than relying on linear correlation alone? |
| Observer independence | Separating chance agreement from true inter-observer consistency | How are independent behavioral coders trained and checked across the study sample using a chance-corrected statistic? |
| Item homogeneity | Checking scale dimensionality and item overlap across survey questions | Do all questionnaire items tap the same underlying construct without repeating identical wording in slightly different phrasing? |
| Construct boundaries | Verifying domain breadth and removing unrelated task demands | Does the instrument cover all core facets of the target construct while avoiding unrelated reading or administration barriers? |
Sources
- OpenStax, Psychology textbook section on Analyzing Findings
- Paul Price, Rajiv Jhangiani, I-Chant Chiang, Dana Leighton, and Carrie Cuttler, Research Methods in Psychology sections on Understanding Psychological Measurement and Reliability and Validity of Measurement
- Institute of Education Sciences Regional Educational Laboratory Midwest, Reliability and Validity Handout
- Mohsen Tavakol and Reg Dennick, Making sense of Cronbach’s alpha, International Journal of Medical Education
- Vikas Menon, Sandeep Grover, Snehil Gupta, and colleagues, A primer on reliability testing of a rating scale, Indian Journal of Psychiatry
- Terry Koo and Mae Li, A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research, Journal of Chiropractic Medicine
- Mary McHugh, Interrater reliability: the kappa statistic, Biochemia Medica
- Godfred Boateng, Torsten Neilands, Edward Frongillo, Hugo Melgar-Quiñonez, and Sera Young, Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research: A Primer, Frontiers in Public Health
- Craig Hedge, Georgina Powell, and Petroc Sumner, The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences, Behavior Research Methods
- Jim Cromwell, Deafness and the art of psychometric testing, The Psychologist
- Dave Bartram, The changing face of testing, The Psychologist
