What Is Differential Item Functioning in an Intelligence Test?

What does the item-level comparison actually compare?

Differential item functioning (DIF) is a difference in the probability of answering a particular test item correctly between groups matched on estimated ability. The focal group is examined against a reference group. The National Research Council’s *Evaluation of the Voluntary National Tests, Year 2: Final Report* says total test score is commonly used for matching. A raw score is the points actually earned. Ability is latent: an underlying trait inferred from responses, not directly observed. Because the estimate depends on responses and a scoring model, “matched” means comparable by the selected measure, not identical in every relevant respect. The matching variable places people at comparable levels; a practical proxy, not proof of directly observed ability. That distinction separates DIF from a simple group score gap. A group’s higher average raw score describes observed test performance. This alone does not show item differences at comparable ability. Similar averages do not establish identical item response patterns. The ETS *Standards for Quality and Fairness* calls raw differences in item percent correct or total scores “impact” and defines DIF in relation to equally able examinees. One summarizes unadjusted performance; the other is conditional on the chosen match. The distinction matters whenever a report uses group differences to discuss a particular item. Consider an example, purely illustrative: a test team wants to examine one reasoning item for two groups. It defines the groups and intended ability. It uses a matching score as a proxy, compares response probabilities at comparable scores, then checks whether the difference warrants review under its method. This describes a group pattern under stated assumptions. It cannot explain why a particular person answered correctly or incorrectly, diagnose that person, rank groups generally, or by itself characterize the whole test. Restate the item, groups, matching measure, and conditional difference; leave cause and practical meaning open to further evidence.

Sources: Evaluation of the Voluntary National Tests, Year 2: Final Report; Standards for Quality and Fairness

Why is a raw score gap a different finding?

A raw score gap and differential item functioning (DIF) describe different results. A raw gap, sometimes called impact, reports an observed difference between groups, such as a difference in average test scores or in the share answering one item correctly. DIF asks a narrower conditional question: among people assigned the same ability-matching score, does a particular item have a different probability of a correct response for the defined groups? The 2014 ETS Standards for Quality and Fairness explicitly distinguish raw differences in item percent correct or test scores from DIF. The National Academies’ Evaluation of the Voluntary National Tests, Year 2: Final Report likewise defines DIF through different probabilities of a correct response after matching examinees on ability. That distinction changes what a result can support. Suppose a report says one group had a lower average score. It describes scores in that sample, not which items behaved differently after matching or why averages differed. A headline that calls the raw gap “DIF” has skipped the matching step. An item-level finding also does not explain the overall gap: a conditional pattern on one item is not an overall mean. The matching score is central. In the National Academies report’s account of voluntary national test development, total test score is generally used as the ability measure; examinees in a focal group are compared with examinees in a reference group who have the same score. Items whose conditional response probabilities differ significantly, or by a specified amount, are flagged for judges to examine. The report also raises practical questions about how the matching criterion is formed and what role DIF statistics should play in item decisions. “Same ability” therefore means matched under the stated score or model, not identical in every relevant way. The reverse inference is unsafe too. Similar group averages do not establish that every item behaves similarly. As a matter of aggregation, different item-level patterns can coexist with similar totals, just as an overall difference can arise without any single item result describing the whole test. This is conceptual, not a claim about a particular intelligence test. The National Academies report’s procedure treats the flag as a prompt for review, and ETS’s glossary separates impact from DIF; neither turns a raw gap or a flag into a complete fairness judgment. When you encounter a claimed finding, check what was compared before accepting its label: group averages, raw item percentages, or item response probabilities after matching? Then ask which groups and matching score were used. A raw gap can matter for interpreting test outcomes, but it cannot alone establish item DIF or unfairness. A reported absence of a raw gap cannot rule out item-specific differences either. Keep the conclusion at the evidence’s level: observed performance and conditional item behavior are separate findings.

Sources: Standards for Quality and Fairness; Evaluation of the Voluntary National Tests, Year 2: Final Report

What does it mean to match people on ability?

To match people on ability is to compare people from defined groups who have similar estimated standing on the ability a test is intended to measure. In differential item functioning (DIF), the question is whether their probability of answering an item correctly differs despite that match. Ability is not directly observed. A latent ability is inferred from response patterns. Analysts therefore need a score or model to represent it, and that choice is part of the evidence.

A common practical match uses examinees’ total test score. The National Academies’ chapter “Technical Issues in Test Development” describes comparing focal- and reference-group examinees with the same score, then flagging items where correct-response probabilities differ significantly or by a specified magnitude. The score is a proxy, not proof that people with equal totals have identical reasoning, knowledge, experience, or circumstances. It establishes a narrower comparison: equal observed standing on this measure, under its scoring and administration.

The tested item contributes to the test whose total may serve as the match. A total can inform estimates of the intended ability, but its meaning depends on what the items sample and how they work together. If a test combines distinct kinds of reasoning, one total may not represent every relevant skill equally. Different response patterns can produce the same sum, and measurement error means an observed score estimates ability imperfectly. These are questions to inspect; they do not show that total-score matching is always invalid.

Researchers must specify which groups are compared and what ability the test is intended to represent. “Same total” does not mean group members are interchangeable in every respect, or establish why an item response differs. The ETS Standards for Quality and Fairness defines DIF in relation to people of equal ability and distinguishes it from impact, or observed group differences in item or test scores. Interpretation depends on whether the chosen match reasonably represents the construct for these examinees and this use.

The study “The Rasch Model Cannot Reveal Systematic Differential Item Functioning in Single Tests” illustrates why analytic choices matter. Its authors examined a 72-item general-reasoning assessment given to Australian Year 10 students, using a Rasch analysis and separate subset analyses. They report that the single-test analysis did not reveal systematic differences identified in verbal and nonverbal subsets. This does not establish a finding about adult IQ tests. A whole-test score may not answer every question about item groups.

When you encounter a DIF claim, ask what score or model matched people, whether it plausibly represents the intended ability for the item, and which population and groups were studied. “Matched on ability” is useful shorthand only when the proxy and assumptions are clear. Equal totals support a conditional comparison on that test; they do not establish equality in every ability, explain an individual response, or settle whether an item is unfair.

Sources: Evaluation of the Voluntary National Tests, Year 2: Final Report; Standards for Quality and Fairness; The Rasch Model Cannot Reveal Systematic Differential Item Functioning in Single Tests: Subset DIF Analysis as an Alternative Methodology

How can the pattern vary across ability levels?

A DIF pattern can stay in one direction across the ability range or change as ability changes. In uniform differential item functioning (DIF), the modeled response curves for two groups remain roughly parallel: at comparable ability, one group has a consistently higher or lower probability of answering correctly. In nonuniform DIF, the gap changes and curves may cross. The direction can then reverse, with one group having the higher modeled probability in one part of the range and the other group elsewhere. These labels describe patterns in an analysis; neither establishes that an item is unfair.

A response curve models the probability of a response at different estimated ability levels. It does not trace a particular person's answers, and it is not directly observed. Its shape depends on the model and available data, so a report should identify the range examined and show the size of the difference there. A summary that says an item favors one group may hide a gap that changes direction across the range. The Osnabrück University MethodsLab explainer notes that crossing curves can make a signed average obscure differences in opposite directions. This clarifies a method; it is not evidence that a particular intelligence test has this pattern.

Imagine an original, non-test situation: two groups receive the same short instructions for assembling a shelf, but one version uses a word more familiar to one group. If that language demand affects groups similarly at lower and higher relevant ability levels, the difference could stay roughly consistent. Now imagine wording that interacts with experience: at one end of the range, familiarity helps one group, while at the other end, how participants use the instructions changes the pattern. The gap might shrink, widen, or reverse. These illustrations show possible shapes; they do not predict how real groups would perform.

So ask more than whether a report says uniform or nonuniform. Where in the estimated ability range does the difference appear? Which group has the higher modeled response probability there, and is the size meaningful for the intended use? The National Research Council’s Evaluation of the Voluntary National Tests, Year 2: Final Report describes DIF analysis as a way to flag items for further judgment. A label alone cannot supply that judgment; model, matching choices, data, item purpose, and content review still matter.

Sources: DIF — Differential Item Functioning; Evaluation of the Voluntary National Tests, Year 2: Final Report

Why do method and data choices matter?

A differential item functioning (DIF) flag is the output of a particular analysis, not a self-explanatory property of an item. Its meaning depends on the sample, group definitions, matching score, model, and uncertainty. A report should make these choices visible. The 1999 National Research Council chapter, “Technical Issues in Test Development,” describes total test score as the usual matching measure and calls for item-fit analysis and further specifications about how DIF results will inform decisions. It also emphasizes human judgment in deciding what a flag means. That account concerns the Voluntary National Tests; it illustrates a workflow, not a universal rule.

Different methods represent the match in different ways. In Mantel–Haenszel analysis, examinees are grouped into strata by total score, which serves as a proxy for ability; response differences between groups are compared within those strata and combined. An item-response approach instead models the probability of a response as a function of ability and compares the modeled patterns for the groups. The university-hosted MethodsLab explainer describes these approaches at an introductory level. Neither description makes the matching variable identical to ability itself: total score is an observed measure used to approximate it. A report should state its score and model, and whether the model fits the data well enough for the comparison. The National Research Council chapter specifically recommends item-fit analyses and asks developers to explain how matching criteria and DIF findings will be handled.

Sparse responses in parts of the score range or one group can make estimates uncertain; reports should show that uncertainty. A statistically detectable difference is not automatically large enough to matter in practice, while a small study may fail to detect a meaningful pattern. When many items are screened, the method for interpreting repeated flags also matters; a count of flagged items alone does not tell a reader whether the whole score is affected. Request analysis details and an effect estimate; no universal sample-size cutoff or p-value replaces judgment.

A published example shows why scope and model must be read together. In a 2021 study of a 72-item general-reasoning test taken by Australian Year 10 students, the authors analyzed a random sample of 1,604 students from a 2019 administration. Their Rasch single-test analyses did not show systematic DIF at the test level, while analyses of verbal and nonverbal item subsets identified subset differences. Subset analysis addresses a different level, not a substitute for item-level analysis. This is one school-age assessment and one analytic approach, not evidence about adult IQ tests generally. It does show why a report should specify what was analyzed. Ask: which method produced this flag, how well did the model fit, how large and uncertain was the difference, and does it persist under a reasonable alternative analysis? A p-value alone cannot answer whether the item is unfair or what to do with it.

Sources: Evaluation of the Voluntary National Tests, Year 2: Final Report; DIF — Differential Item Functioning; The Rasch Model Cannot Reveal Systematic Differential Item Functioning in Single Tests: Subset DIF Analysis as an Alternative Methodology

When is a single-item result too narrow?

A differential item functioning (DIF) result is usually about one item: at a stated level of estimated ability, does response probability differ between defined groups? That item-level finding cannot by itself tell you how much any difference changes a total score, whether a whole scale works equivalently across groups, or whether scores support the same interpretation. A score combines many items; a flag isolates one conditional comparison. Four claims are easy to blur. An item flag identifies a difference for a specified item under a specified model and match. A subset analysis asks whether a defined collection of related items behaves differently against the broader assessment. Aggregate differential test functioning asks about the combined effect across a test. Measurement invariance is a broader question about whether measurement can be compared across groups. These questions are related, but none is a synonym for another. The National Research Council’s *Evaluation of the Voluntary National Tests, Year 2: Final Report* describes DIF as an item-level comparison after matching on ability, with statistical flags followed by review. That workflow does not make one flagged item a finding about a complete score or scale. If several items show differences in the same direction, their effects could contribute to an aggregate score difference. If item differences point in opposing directions, they could partly offset in the total. These are logical possibilities, not claims about a particular intelligence test. A whole-test summary may describe the combined score while obscuring which subsets contributed. Conversely, a difference in one subset does not establish that the total score is unusable; size, direction, content and intended use still matter. A published example shows why a report should name its analytic level. In “The Rasch Model Cannot Reveal Systematic Differential Item Functioning in Single Tests: Subset DIF Analysis as an Alternative Methodology,” researchers examined a 72-item general reasoning assessment with verbal and nonverbal subsets. The data came from a random sample of 1,604 Australian Year 10 students drawn from a 2019 administration. In Rasch analyses treating the subsets as single tests, observed means followed the model’s test characteristic curves, which the authors interpreted as no systematic DIF at that level. Separate subset analyses, using the combined assessment as a frame of reference, found different patterns: the nonverbal subset favored boys , while the verbal subset favored girls. The authors describe subset DIF as testing a different assumption, not as a direct substitute for systematic item-level DIF analysis. This is a methodological case from one school-age assessment and one analytic approach. It does not establish a result for adult IQ tests, quantify how often DIF occurs, or prove that whole-test analysis is generally inadequate. conclusions depend partly on what was grouped and compared. When a report moves from a flagged item to a claim about a subset, full score, or test fairness, check whether broader evidence was presented. Ask whether the finding concerns one item, a predefined cluster, or the combined score, and what supports the larger conclusion.

Sources: The Rasch Model Cannot Reveal Systematic Differential Item Functioning in Single Tests: Subset DIF Analysis as an Alternative Methodology; Evaluation of the Voluntary National Tests, Year 2: Final Report

Why does a DIF flag require content review?

A DIF flag is a reason to inspect an item, not a diagnosis of its cause. It means modeled response probabilities differed between groups matched on an ability measure. It does not reveal whether the difference comes from an irrelevant obstacle, a legitimate part of the ability being measured, administration conditions, or limitations in the analysis. The National Research Council’s Evaluation of the Voluntary National Tests, Year 2: Final Report says flagged items should be examined by judges to evaluate the differences. It also calls for specifications explaining how DIF results will inform item decisions. A statistical pattern and an editorial judgment answer different questions.

Review begins by stating the test’s intended construct precisely: the skills or knowledge the test is supposed to measure, including what it deliberately excludes. The ETS Standards for Quality and Fairness says test validation starts with a clear account of the construct, purpose, intended score interpretations, and target population. Reviewers can then ask whether an item demand belongs to that definition. For example, imagine a reasoning item whose instructions rely on uncommon vocabulary. If the intended ability is nonverbal pattern reasoning, that vocabulary may add a reading demand outside the stated construct. In a reading assessment, the same kind of language demand could be central to what the item is designed to measure. This illustration does not establish that either item would show DIF; it shows why a demand cannot be judged apart from the test’s purpose.

Context and delivery also deserve inspection. The ETS International Principles for the Fairness of Assessments gives the example of a diagram whose small labels may impede a visually impaired test taker, even when the tested skill involves interpreting the diagram. An unfamiliar setting may also affect access, but that possibility alone does not explain a response difference. Reviewers should inspect the item, instructions, administration conditions, and intended population, then distinguish evidence about the item’s demand from a plausible but untested explanation.

A bounded review can follow four steps: inspect the item and administration; consider plausible explanations and what evidence could distinguish them; ask qualified content and measurement reviewers to assess the item against the test specifications; and document whether it is retained, revised, or removed, with reasons and consequences for content and score interpretation. The ETS Standards for Quality and Fairness calls for qualified reviews of items and related materials, documenting the review’s purpose, process, results, and reviewer qualifications. The National Research Council report likewise asks test developers to specify how statistical results and expert judgment will inform item decisions.

Review does not prove every remaining item fair or every group and use adequately studied. Retention may be defensible when the demand belongs to the construct and broader evidence supports the use. Revision or removal may be appropriate when evidence points to an irrelevant barrier. Ask what feature reviewers examined, what ability the test intended to measure, and how evidence shaped their decision.

Sources: Evaluation of the Voluntary National Tests, Year 2: Final Report; Standards for Quality and Fairness; International Principles for the Fairness of Assessments

What does a published general-reasoning example add?

A published case shows why the scope of a DIF conclusion must match the analysis that produced it. In “The Rasch Model Cannot Reveal Systematic Differential Item Functioning in Single Tests: Subset DIF Analysis as an Alternative Methodology,” researchers examined one general-reasoning assessment used with Australian Year 10 students. The findings illustrate how whole-test and item-subset analyses can make different patterns visible. They do not establish how intelligence tests generally behave or settle another test’s fairness. The assessment contained 72 dichotomous items: 36 nonverbal reasoning items and 36 verbal reasoning items. The study used a random sample of 1,604 students, average age about 15, drawn from a 2019 administration across 12 secondary schools. This is a defined school-age sample and a particular test, not a representative sample of adults taking IQ tests. The comparison groups were boys and girls; the results are not rankings or claims about individuals. The authors first analyzed the verbal and nonverbal sets as single tests using the Rasch model. In these analyses, observed group means followed the expected score curves, so the authors reported no systematic DIF at that level. They then conducted their proposed subset analyses, using person estimates based on the full test as a frame of reference for comparing performance within each subset. Those analyses found subset DIF: the nonverbal subset favored boys in the authors’ analysis, while the verbal subset favored girls. The pattern changed with the analytic question and reference frame. A single-test analysis and a subset analysis examine different properties. In their discussion, the authors explain that the Rasch total score determines a person’s estimate within a single test; this constrains the whole-test residual pattern they examine. Their subset method instead asks whether group performance within components is consistent with estimates formed using the broader set of items. They explicitly say subset DIF analysis is not a direct alternative to systematic item-level DIF analysis: it offers insight into patterns across item clusters. The distinction matters because a whole-test result cannot automatically answer every question about its components. The study is a methodological example, not a universal verdict on Rasch analysis or a template that every intelligence test should follow. Its conclusion depends on the model, item groupings, data, and proposed procedure. The article itself notes that other forms of model misfit may complicate interpretation and that selecting a frame of reference can affect what is being measured. Its findings do not show that all whole-test DIF analyses miss subgroup patterns, nor that a subset flag proves an item is unfair. When a report cites one analysis as proof that an entire intelligence test is fair or biased, ask what level it examined. Did it analyze particular items, a planned subset, or a total score? What method and reference frame were used, and what patterns could that method detect? This case makes the practical point precise: a result can be informative within its stated scope while leaving other score and item questions unresolved.

Sources: The Rasch Model Cannot Reveal Systematic Differential Item Functioning in Single Tests: Subset DIF Analysis as an Alternative Methodology

Which fairness questions sit beyond item-level DIF?

Differential item functioning (DIF) asks whether a particular item behaves differently for defined groups after people are matched on the ability the test is intended to measure. It does not answer whether a test score predicts an outside outcome similarly across groups, supports a particular decision, or explains one person's answer. Those are separate questions requiring evidence suited to their uses. In employment selection, researchers may ask whether cognitive test scores relate to later job performance similarly across groups. The Annual Review article “Differential Validity and Differential Prediction of Cognitive Ability Tests: Understanding Test Bias in the Employment Context” reviews these score-to-outcome questions. Its abstract reports evidence of differences in correlations, slopes, and intercepts between White and non-White subgroups. This is a review focused on employment. It does not show that a particular item has DIF or establish how every intelligence test performs. The distinction is the level of analysis: DIF concerns a conditional response to an item; differential prediction concerns how scores relate to an external criterion. Neither result substitutes for the other. A group-level item analysis cannot diagnose a person, explain their response, or establish their ability from one answer. Nor does a reported absence of DIF certify every item or use: it is bounded by the groups, sample, items, matching measure, and model examined. The National Academies report “Evaluation of the Voluntary National Tests, Year 2: Final Report” describes DIF as comparing item-response probabilities after matching on ability, typically using total test score, then reviewing flagged items. This procedure informs an item-level investigation; it does not establish that the whole score supports a particular interpretation or decision. For consequential education, clinical, or employment decisions, ask what evidence supports the score for that specific purpose and population, and consult a qualified assessment professional. Test IQ Free’s browser test is for low-stakes reasoning practice. Its displayed IQ range is a provisional estimate from an unvalidated conversion with no validated adult reference population; it is neither a normed comparison nor a confidence interval. No DIF study of its item pool is claimed. The optional $9 report adds explanation, not validity. When discussing a specific DIF claim with a provider or examiner, ask what external outcome was studied separately, if any, and what that evidence permits you to conclude.

Sources: Differential Validity and Differential Prediction of Cognitive Ability Tests: Understanding Test Bias in the Employment Context; Evaluation of the Voluntary National Tests, Year 2: Final Report

What should you ask when a report says an item showed DIF?

When a report says an item showed differential item functioning (DIF), ask which item and groups were compared, how ability was matched, and the finding. The National Research Council’s Evaluation of the Voluntary National Tests, Year 2: Final Report describes matching on total scores and comparing response probabilities. A DIF finding concerns groups, not why one person answered as they did.

Ask how large and uncertain the difference was, which analysis produced it, and what limits apply to its matching score and data. A flag alone does not show practical importance. Ask whether reviewers considered the item’s wording, context, administration, and intended reasoning demand. The National Research Council report calls for human judgment in item decisions; a flag does not automatically mean removal.

Keep the conclusion at the level studied: evidence about one item does not establish that a test is biased, fair, or suitable for a decision. For a consequential decision, ask a qualified examiner: “What does this analysis support about the item, and what further evidence supports using the full score for my purpose?”

For practice, use Test IQ Free’s free 50-question test and worked answers. Its displayed IQ range is a provisional estimate from an unvalidated conversion with no validated adult reference population, not a normed comparison or confidence interval. The paid report adds explanation, not validity.

Sources: Evaluation of the Voluntary National Tests, Year 2: Final Report

Questions readers ask