On this page
Reliability and validity are separate properties. Reliability measures consistency, whether a measurement produces the same result repeatedly. Validity measures accuracy, whether it actually captures what it claims to measure. A measurement can be reliable without being valid, or valid without being reliable. Research requires both.
- Reliability is consistency of measurement over time, across raters, or across items; validity is accuracy of what is being measured.
- A reliable measurement produces consistent results but may be consistently measuring the wrong thing.
- A valid measurement captures the right construct but may do so inconsistently across administrations or observers.
- Researchers assess reliability through test-retest correlation, inter-rater agreement, and internal consistency statistics like Cronbach's alpha.
- Researchers assess validity through construct validity evidence, criterion validity correlation, and content validity review by subject matter experts.
Research findings are only as trustworthy as the measurements behind them. Two properties determine whether a measurement can be trusted: reliability and validity. These terms are often used interchangeably in casual speech, but they describe fundamentally different things. A measurement can be reliable without being valid, or valid without being reliable. Understanding the distinction matters because each failure undermines research in a different way, and each requires different corrective steps. A researcher who conflates them may believe a study is sound when its core measurements are flawed.
Why reliability and validity are not the same thing
Reliability and validity answer two separate questions about measurement. Reliability asks: Does this measurement produce consistent results? Validity asks: Does this measurement actually capture what it claims to measure? A measurement can satisfy one criterion and fail the other. A broken thermometer that always reads 98 degrees is reliable, it is consistent, but invalid, because it does not accurately measure temperature. A mood scale that genuinely captures emotional state but produces slightly different scores each time it is administered is valid but unreliable. Confusing these properties leads researchers and readers to trust measurements that are fundamentally flawed. The distinction is not semantic; it is structural. Reliability is a precondition for validity, but it is not sufficient. Consistency alone proves nothing about accuracy.
What reliability means in research
Reliability refers to the consistency of a measure. It is the consistency of a measurement, whether a scale, test, coding scheme, or instrument produces the same result under the same conditions. If a depression screening tool is reliable, it should yield similar scores when the same person takes it twice within a short interval, assuming their depression has not changed. If a coding scheme is reliable, different researchers applying it to the same text should reach the same conclusions. Reliability is necessary because measurement error obscures the true signal. Without consistency, a researcher cannot distinguish between real variation in what is being measured and noise from the measurement process itself. But consistency does not prove the measurement is capturing the right thing.
Types of reliability researchers assess
Psychologists consider three types of consistency in reliability: over time (test-retest reliability), across items (internal consistency), and across different researchers (inter-rater reliability). Test-retest reliability examines whether a measurement produces similar results when administered repeatedly. It assesses whether a test or measurement tool produces similar results when administered to the same people on two or more occasions, assuming the underlying trait being measured has not changed. Internal consistency reliability is an estimate of reliability based on the average correlation among items within a test. If a 10-item anxiety scale is internally consistent, the items correlate with each other, suggesting they all measure the same construct. Inter-rater reliability measures whether different observers or coders applying the same measurement scheme reach the same conclusions. A diagnostic checklist has high inter-rater reliability if two clinicians using it on the same patient produce consistent assessments.
What validity means in research
Validity refers to how accurately a method measures what it is intended to measure. If research has high validity, it produces results that correspond to real properties, characteristics, and variations in the physical or social world. A valid measurement captures the construct it claims to measure, not something else that happens to correlate with it. A test of mathematical ability should measure mathematical reasoning, not reading comprehension or test-taking familiarity. A biomarker for disease severity should reflect actual disease burden, not patient age or treatment history. Validity is about accuracy: the measurement reflects reality. This is why a measurement can be reliable but invalid. A scale can consistently produce the same scores without those scores meaning what researchers claim they mean.
Types of validity researchers assess
Construct validity evaluates whether a measurement tool really represents the thing we are interested in measuring. It asks whether scores on an instrument genuinely reflect variation in the construct, or whether they reflect something else that happens to correlate with it. Construct validity evidence types include convergent, discriminant, nomological, known-groups, and factorial validity. Convergent validity is demonstrated when a new measure correlates strongly with existing measures of the same construct. Discriminant validity is shown when a measure does not correlate with measures of different constructs. Criterion validity assesses how a new scale correlates with a criterion or gold standard. Depending on the time of administration of the gold standard, this can be classified as concurrent or predictive validity. A new depression screening tool has criterion validity if it correlates with clinician diagnosis (concurrent) or predicts future depressive episodes (predictive). Face validity is the extent to which a measure appears to measure what it is supposed to measure at face value. It is a subjective assessment by experts based on experience about whether the measure reflects its intended assessment, and is the weakest form of validity testing because it is not based on objective observation.
How a measurement can be reliable but not valid
A measurement can be highly consistent and still measure the wrong thing. Suppose a researcher wants to measure intelligence but uses a test that actually measures reading speed. If the test is well-designed and produces consistent scores, it is reliable. But it is not valid as a measure of intelligence, because it is capturing reading ability instead. The measurement is precise and reproducible, it fails on accuracy. Consider a workplace stress survey that asks employees how many hours they work per week. The responses would be highly reliable; people can report their hours consistently. But hours worked does not validly measure stress. Someone working 50 hours might be stressed or energized; someone working 30 hours might be overwhelmed or content. The measurement is reliable but invalid because it does not capture the construct it claims to measure. This is why a study can report high internal consistency or strong test-retest correlation and still be fundamentally flawed. The measurement is doing its job consistently; it is just doing the wrong job.
How a measurement can be valid but not reliable
A measurement can capture the right construct but do so inconsistently. Suppose a clinical assessment tool genuinely measures patient pain severity, it asks the right questions and interprets responses correctly. But the tool is administered in different ways by different clinicians: some spend five minutes, others spend 20 minutes; some probe follow-up responses, others do not. The same patient might report a pain score of 6 with one clinician and 8 with another, not because their pain changed but because the administration was inconsistent. The measurement is valid, it targets the right construct, but unreliable because the process is not standardized. Or consider a diagnostic imaging test that accurately identifies disease when read carefully but produces inconsistent results because the image quality varies or the reader is fatigued. The test is measuring what it claims to measure, but the measurement process is not stable. A measurement can be valid but unreliable when the underlying construct is real and the measurement targets it, but the execution is inconsistent. This is why researchers must assess both properties separately. Validity without reliability means the measurement is aiming at the right target but missing it sometimes.
How researchers test for both reliability and validity
Researchers use specific statistical methods and study designs to assess each property. For reliability, the method depends on the type of consistency being evaluated. The appropriate measure for reporting internal consistency is Cronbach's alpha (α). For test-retest reliability, the appropriate measure is the intraclass correlation coefficient (ICC) for continuous variables and intraclass kappa for categorical variables. Test-retest reliability operates through three basic steps: first, the same test must be administered to the same group of individuals at two different times; next, the correlation coefficient between the scores of the two assessments must be calculated; the final step is to interpret the correlation coefficient. For validity, researchers use different approaches depending on which type they are assessing. Convergent validity and discriminant validity are tested with correlations to see if results from a test are positively or negatively related to those of other established tests. Convergent construct validity is assessed by the statistical correlation between scores on a single health component, as measured by two different instruments, and if the correlation coefficient is positive and appreciably above zero, the new measure is said to have convergent construct validity. Both exploratory and confirmatory factor analysis techniques provide vital evidence to support construct validity. Criterion validity is tested by correlating the new measure with an accepted gold standard. These are not one-time checks; they are ongoing assessments that accumulate evidence.
Why both matter for research quality
A study's findings are only trustworthy if its measurements are both reliable and valid. If a measurement is unreliable, the results are noisy and unstable; the same study run again might produce different conclusions. If a measurement is invalid, the results are systematically wrong; the study is answering a different question than the one it claims to answer. Unreliability introduces random error that obscures the true effect. Invalid measurement introduces systematic bias that points researchers in the wrong direction. A researcher who reports high internal consistency but has not established validity may have a precise instrument that measures nothing of interest. A researcher who shows criterion validity but has not tested reliability may have a measurement that works in one context but fails in another. Both properties must be demonstrated for readers to have confidence in the findings. When evaluating research, ask whether the authors have reported reliability statistics appropriate to their measurement method and whether they have provided evidence of validity through correlations with established measures, expert review, or predictive accuracy. If either is missing, the measurement foundation is incomplete, and the conclusions rest on uncertain ground.
Do this automatically
Researchers manually assessing reliability across test-retest cycles, internal consistency checks, and inter-rater comparisons must repeat coding schemes and statistical calculations for each participant, each item, and each evaluator, a process that multiplies in complexity as sample sizes and measurement dimensions grow. Software can systematically apply these consistency checks and validity assessments across all data points simultaneously, flagging where measurements diverge from their intended constructs. Academia research runs this kind of analysis end to end and cites every figure back to its original source, across as many subjects as you need, with Mimir attaching a citation to every figure. It will not write your paper for you. It gathers and cites the evidence; the argument stays yours, and deliberately so. Every plan includes it, including the free one.
Start your first report and run it on something you already know well.
This article is general information and research commentary. It is not professional advice, and it does not account for your particular circumstances.
