What Is an Item Discrimination Index in an Intelligence Test?

What does an item discrimination index mean?

An item discrimination index describes how an item’s responses relate to a stated comparison, such as performance on the rest of a test or membership in higher- and lower-scoring groups. It is evidence about how that item behaves under that comparison; by itself, it is not a measure of a person’s intelligence and does not establish what a test score means. The label is a family name, not a complete method. An upper/lower index contrasts the item’s response proportions in selected score groups; an item-rest correlation relates responses to a score formed from the other items; an IRT discrimination parameter is estimated within a model of response probability. For the common upper/lower convention, the sign guardrail is D = pU − pL, where the upper-group proportion comes first. Reports can use one label consistently within a platform, but conventions are not universal. Before interpreting a number, identify its formula or model, comparison group, and response coding; two values called “discrimination” may answer different questions and cannot be compared safely from the label alone. The relevant conclusion is conditional: an item separates responses according to the chosen criterion to some degree, not that it distinguishes intelligence in general.

Sources: Understanding Item Analyses; Alternative Methods for Item Parameter Estimation: From CTT to IRT

How is the upper/lower difference calculated?

For a binary-scored item, the usual upper/lower difference is the proportion answering correctly in the selected higher-scoring group minus the proportion answering correctly in the selected lower-scoring group: D = pU − pL. Here, pU is the fraction correct in the upper group and pL is the fraction correct in the lower group. The University of Delaware’s “Item Analysis of Classroom Tests: Aims and Simplified Procedures” gives this direction explicitly by subtracting the lower-group correct count from the upper-group count and dividing by the group size. Equivalently, when both groups have the same size, subtract their correct-response proportions. D is a difference in proportions, often written as a decimal or percentage-point difference; it is not the percentage of all examinees who answered correctly. A positive D means the upper group had a higher correct proportion under this grouping and coding. Zero means the observed proportions match. A negative D means the lower group had the higher proportion, which merits checking the item and scoring before drawing a conclusion. None of these signs alone diagnoses why the pattern occurred.

For an explicitly hypothetical calculation, suppose the selected upper group has 18 correct responses among 24 people, while the selected lower group has 10 among 24. Then pU = 18/24 = 0.75 and pL = 10/24 ≈ 0.42, so D ≈ 0.33, or about 33 percentage points. Reversing the subtraction would produce −0.33 and falsely describe the direction under this convention. The numbers illustrate arithmetic only; they are not study results or recommended cutoffs. If both groups had 18 of 24 correct, D would be zero. If the upper group had 10 and the lower 18, D would be about −0.33.

The score used to form the groups is part of the statistic’s meaning. A report should say whether examinees were ordered by the total test score, a domain score, or another criterion, and how many from each end were retained. A classroom procedure in the University of Delaware guidance illustrates selecting the high and low groups and leaving the middle out; this is one procedure, not a universal cutoff rule. If group sizes differ, calculate each proportion using its own denominator before subtracting. The report should also disclose how ties at a boundary were handled: including all tied examinees can change group sizes, while a fixed-size rule may split or otherwise resolve a tie. For a multi-part or partially credited item, the binary correct/incorrect interpretation may not apply, so the scoring and statistic need to be identified. Extreme groups make the comparison easy to explain, but they discard middle observations and make the result depend on selection choices. To interpret or compare D, readers need the direction, criterion, group definition, cutoff, tie rule, and item-scoring convention—not just the symbol.

Sources: Item Analysis of Classroom Tests: Aims and Simplified Procedures

Why can the cutoff change the number?

A 27% upper/lower split is a historically influential convention, not a universal evidence-based optimum. Changing the cutoff changes who counts as “upper” and “lower,” and therefore changes the item-response proportions being contrasted. More extreme groups can make the contrast sharper by separating examinees farther apart on the reference score, but they also use fewer people. That smaller base can make estimates less stable and leaves more of the sample outside the comparison. A less extreme split brings more examinees into the groups, including people whose scores are closer together, which can soften the contrast. The direction and size of the resulting change depend on the response distribution and the item; “more extreme” does not mean “more accurate” in every setting. The choice is part of the statistic’s definition, not a cosmetic reporting detail.

“Dimension-corrected discrimination index” examined 27% and 20% upper/lower cutoffs using a training dataset and external datasets. The authors report that cutoff choice could produce underestimation or overestimation relative to their comparison, with the pattern varying across datasets. One external dataset was a national grade-9 mathematics assessment with 11,235 examinees; the paper also used other external data. These are analyses of those datasets and item pools, not intelligence-test norm samples. They show why a cutoff result cannot be assumed to transport unchanged to a different population or test. The authors call for systematic validation, so the comparisons are evidence that the convention can matter, not a settled replacement rule for intelligence testing.

The paper also separates binary-scored from polytomous items rather than treating them as interchangeable. Its reported cutoff behavior differed by item response format: a rule evaluated on right-or-wrong responses did not simply carry over to items that can receive multiple score values. Difficulty mattered as well. Across the analyses, the relationship between cutoff and the reported index varied with item difficulty, so a single adjustment could not be read as working uniformly across easy and difficult items. This matters because an apparent difference between two item indices might reflect a change in group definition, scoring format, or difficulty mix, as well as a change in response behavior. The paper’s results concern its particular datasets and methods; they do not establish a universal direction or correction for every item.

The historical rationale for 27% still matters: “A Rationale for Evaluation of Item Discrimination Statistics” is an earlier source for the convention’s mathematical argument under particular assumptions about score distribution and item difficulty. The newer findings do not erase that rationale. They qualify what follows from it: a rationale under specified conditions is not proof of a universal empirical advantage across formats, difficulty levels, populations, and tests. Nor does the newer paper establish that 20%, or any proposed correction, should replace 27% in intelligence testing. Until a report documents a supported local method, readers should treat the cutoff as a condition on the result, not as a best-practice badge.

For a meaningful comparison, record the actual upper and lower cutoff, how boundary ties were handled, the item’s response format and scoring, and the reference score used to order examinees. Compare values only when those choices and the relevant sample conditions are sufficiently alike; otherwise, the same label may conceal different contrasts. If the report does not state its cutoff or whether responses were binary or polytomous, ask for the formula and scoring rule before interpreting a difference. This is the practical consequence of the evidence: cutoff selection can alter the number, while available findings do not authorize one universally preferred split for intelligence-test items.

Sources: Dimension-corrected discrimination index; A Rationale for Evaluation of Item Discrimination Statistics

What does item-rest correlation compare?

An item-rest correlation relates each examinee’s response to a particular item to that same examinee’s score on the remaining items. “Rest” means the target item is excluded from the comparison score. In “Understanding Item Analyses,” the University of Washington describes its ScorePak statistic as a Pearson correlation between an item response and scores on all other items. For a binary item, responses may be coded as correct or incorrect; the rest score is calculated from the other scored items. Across examinees, the correlation summarizes whether people with higher rest scores tend to have higher coded responses on the item, lower responses, or no consistent linear association. The sign depends on coding: if correct is coded higher, a positive association means higher rest scores tend to accompany correct answers, while a negative association means the reverse pattern in that sample. It does not mean the item itself has a positive or negative moral or educational quality.

The exclusion distinguishes a corrected item-rest correlation from an uncorrected item-total correlation. If the “total” includes the target item, the item response contributes to both sides of the correlation. That shared part can raise the observed association mechanically: the statistic partly relates the item to itself as well as to the rest of the test. Removing the target item from the comparison score avoids that direct part-whole overlap. A report may call its statistic “item-total” even when it uses a corrected total, or it may use another implementation; the label alone does not settle the formula. Check whether the target response is included in the comparison score, how responses are coded, and which items compose the rest score.

This correction does not make item-rest correlation automatically superior or more valid. It still depends on the examinees included, the other items that make up the rest score, and whether that score represents one coherent dimension or combines different abilities or content. A positive association therefore describes alignment with that particular remaining-item score in that sample; it does not establish that the item measures intelligence generally or that the overall test supports a proposed interpretation. The useful reading question is precise: which response was correlated with which score, calculated how? If a report does not answer that, request the computation before comparing its number with another report’s item-rest or item-total value. The same questions apply when comparing versions of a test: if items were added, removed, or reassigned to domains, the rest score itself changed, so the comparison criterion is no longer identical. That is a reason to inspect the score definition and item set, not to assume that one coefficient can be carried unchanged into the new form.

Sources: Understanding Item Analyses

Why can classical indices disagree?

They can disagree because the upper/lower difference and item-rest correlation retain different information from the same response sheet. The upper/lower difference compares correct proportions among examinees selected from opposite ends of a reference score and leaves the middle out. The item-rest correlation instead summarizes how an item response covaries with the remaining-item score across the examinees included in its calculation. The University of Delaware’s “Item Analysis of Classroom Tests: Aims and Simplified Procedures” makes the selected-group procedure explicit; the University of Washington’s “Understanding Item Analyses” defines its ScorePak measure as a correlation with the score on all other items. Thus one statistic is a contrast between chosen groups, while the other is an across-sample association. Neither is simply a more detailed version of the other.

Consider an invented example, solely to show the arithmetic logic. Suppose a hypothetical set has 30 examinees: the 10 in the upper group and 10 in the lower group each have 8 correct answers on a target item, while the 10 middle examinees are excluded from D. D is therefore 8/10 minus 8/10, or zero. Now keep those two groups and their item responses unchanged, but imagine that the middle group’s target responses shift from five correct to zero correct, while their remaining-test scores stay fixed. The upper and lower proportions have not moved, so this extreme-group difference remains zero. Yet the item’s association with remaining-test scores across all 30 examinees can change, because the middle responses have changed relative to their comparison scores. All numbers in this invented pattern are illustrative, not findings from a study or observed examinees. The point is structural: observations ignored by one summary can contribute to the other.

That example does not predict which direction a real correlation will move; the result depends on how item responses line up with the remaining scores across the full pattern. Nor does disagreement show that one calculation is wrong. Each statistic answers an operationally different question: how sharply do selected extremes differ on this item, or how does the item response associate with the rest score across the analyzed sample? Scoring errors, group-selection rules, sample composition, response coding, or a poorly representative rest score can affect either summary. When the values conflict, inspect the underlying responses and definitions before treating either number as a verdict on item quality.

Sources: Item Analysis of Classroom Tests: Aims and Simplified Procedures; Understanding Item Analyses

What changes when a model estimates discrimination?

An IRT discrimination parameter is an estimate inside a model of how response probability changes along a modeled ability scale. That makes it a different kind of quantity from an upper/lower D or an item-rest correlation. D is calculated from observed correct-response proportions in selected groups; item-rest r summarizes an observed association between item responses and remaining scores. An IRT parameter is inferred by fitting a specified response model to item and examinee response data. ETS’s “Alternative Methods for Item Parameter Estimation: From CTT to IRT” explicitly treats item-parameter estimation across classical test theory (CTT) and item response theory (IRT), anchoring this distinction between observed-score summaries and model-based estimation. The three labels may all be discussed as discrimination, but they do not share a common formula or automatic scale of interpretation.

At an intuitive level, the model asks how likely a response is at different positions on its latent ability continuum, while accounting for the response patterns represented in the calibration data. The discrimination parameter describes a modeled relationship in that conditional response function; it is not a count of how many people in an observed subgroup got an item right, and it is not a correlation computed directly against a remaining-item total. Its meaning therefore depends on what response model was fitted, how the ability scale was defined, what items and examinees informed calibration, and whether the model adequately describes the responses. Without those details, a reported parameter cannot be read as a portable measure that can be compared directly with D or r.

The distinction is also about what is held constant in the question. A classical summary describes the observed sample under its chosen score comparison; an IRT analysis uses a model to express response probability conditional on a latent position. That conditional description depends on the model’s assumptions and on the calibration information available to estimate it. If two reports use different models, ability scales, calibration samples, or item pools, equal-looking parameter labels need not denote interchangeable quantities. A classical value can be reproducible as a sample calculation yet remain sample-dependent; a model estimate can be useful within its calibrated framework without becoming a universal item property.

Calibration context matters because estimates reflect the design and method used to obtain them. In “Practical Considerations in Item Calibration With Small Samples Under Multistage Test Design: A Case Study,” ETS examined an operational multistage-testing program alongside program-specific simulations and compared calibration approaches. The report found performance depended on method and sample design in those conditions; its case involved mostly easy items and the authors restrict generalization. This is an example of why a model parameter’s reported value belongs with information about the calibration sample and procedure. It does not establish a general sample-size threshold, nor does it show that one calibration method is best for every intelligence test.

A more elaborate model is not automatically more accurate or more portable. Model fit concerns whether the chosen response model is adequate for the data and purpose; calibration concerns which examinees, items, and response patterns informed the estimates; the scale defines what a unit of modeled ability means within that analysis. Item pool and intended use also matter when deciding whether a parameter supports the interpretation being proposed. ETS’s CTT-to-IRT overview supports distinguishing estimation frameworks, while the multistage calibration case shows that method and design can affect results in a particular program. Together, they support a reporting rule, not a universal ranking: identify the model and scale, describe the calibration context and method, and provide relevant fit information before interpreting or comparing an IRT discrimination estimate.

Sources: Alternative Methods for Item Parameter Estimation: From CTT to IRT; Practical Considerations in Item Calibration With Small Samples Under Multistage Test Design: A Case Study

What can a high or low value establish?

A discrimination value is useful as a description of how an item’s observed responses align with the comparison built into that statistic. A positive association can show that, in the analyzed sample and under the stated scoring, people who did better on the reference score also tended to answer this item correctly. That is evidence about response behavior relative to that criterion. It does not by itself show that the item measures intelligence broadly, works equally well for every group, or supports a particular decision about a person. The University of Washington’s “Understanding Item Analyses” describes its ScorePak data as a tentative diagnostic aid affected by the examinees and administration; that is a useful way to read such results: a clue about this use and sample, not a permanent grade attached to the item.

Difficulty and discrimination answer different questions. The University of Washington’s “Understanding Item Analyses” defines difficulty for a dichotomously scored item as the percentage correct. If nearly everyone answers an item correctly, or nearly everyone misses it, the item may offer little variation for separating examinees under a chosen association measure. A lower index in either case can follow from the response distribution; it does not alone prove poor wording, bad scoring, or lack of value. Conversely, an item with an intermediate success rate is not automatically strong: its responses still need to align with the relevant comparison, and its content must serve the test’s purpose. A difficulty statistic says how often the item was answered correctly in that group; a discrimination statistic summarizes a particular relationship among responses. Neither substitutes for the other.

Content coverage adds another reason an item-level association may be modest. A test may intentionally sample several reasoning operations or domains, while a total score combines performance across them. An item representing a less common but required content area can align less closely with that combined score than an item repeating the dominant format. The University of Washington’s “Understanding Item Analyses” notes that items sampling different content may have lower associations with the total and still be needed. This does not establish that any low-association item is worth retaining. It means the reviewer must ask whether the item is accurate and relevant to the intended construct, and whether removing it would narrow what the test covers. A statistic that rewards alignment with the existing total can flag unusual behavior while remaining unable to decide whether the content belongs.

A negative value deserves attention because, under the report’s coding and criterion, responses run opposite to the expected direction. It can indicate a reversed or incorrect key, coding error, confusing wording, a distractor that attracts higher-performing examinees, or a mismatch between the item and the comparison score. The University of Washington’s “Understanding Item Analyses” recommends using response patterns to investigate item performance, and “A Rationale for Evaluation of Item Discrimination Statistics” discusses how wrong-option patterns can contribute information about distractors. Those sources do not make a negative result harmless: if the key is wrong or examinees interpret the wording unexpectedly, the item may materially undermine the score. But the sign is a reason to inspect the item, key, coding, distractors, response data, and content fit—not an automatic instruction to delete it. A deletion decision should follow the diagnosed cause and the item’s role in the test.

The boundary is validity. The joint AERA, APA, and NCME “Standards for Educational and Psychological Testing” frame validity as evidence and theory supporting the interpretations made from scores and the uses proposed for them. In that framework, validity is not a badge bestowed by one coefficient or by the test name. An item association may contribute evidence about how responses relate to a defined score, but it cannot alone establish that the overall score represents the intended construct, that the interpretation applies to a particular population, or that using the result for a decision is justified. Those claims need evidence matched to the interpretation and use, including relevant content and response evidence and other sources of support. The Standards provide a framework for that argument; they do not certify any particular test or index.

This distinction matters because an item can behave consistently with a total score that is itself a poor basis for the intended conclusion. A high association may reflect overlap with the rest of a narrow test, a shared format, or the sample’s particular response pattern. It cannot rule out omitted content, construct underrepresentation, or a mismatch between the score and the decision being made. A lower association may reflect noise or a defective item, but it may also flag content that the total represents less strongly. Item analysis helps locate questions for review; it does not settle the meaning of the test score. To move from item behavior to a claim about an intelligence score, evidence must connect the test’s content and scoring to the interpretation for the examinees and context at issue.

The practical judgment is therefore neither “keep every low item” nor “remove every low item.” Treat a markedly low or negative result as a potentially serious warning, then trace it to the responses and the item: verify the key and coding, examine wording and distractor choices, check whether administration differed, and consider whether the content fits the intended domain. Compare that diagnosis with the rest of the evidence for the score interpretation and use. If the result persists across relevant examinees and conditions and points to a defect, revision or removal may be warranted; if it reflects necessary content coverage, the response may be to improve the item or report its limitation. The index narrows the review question. The evidence about content, score meaning, and proposed use determines what follows.

Sources: Understanding Item Analyses; Standards for Educational and Psychological Testing

How should a reader check an item-analysis report?

Read a reported value in three steps: identify the formula or model and its comparison; note who took the test, how it was administered and coded, and what the item was intended to cover; then ask what interpretation or decision the index is meant to support. If those details are missing, request them before comparing the number or acting on it. For low-stakes practice, Test IQ Free offers a free 50-question reasoning test and worked answers. Its displayed estimated IQ range is provisional, produced by an unvalidated conversion with no validated adult reference population. The optional $9 report adds explanation and practice structure, not validity or accuracy. For consequential educational, clinical, or other high-stakes decisions, use an appropriately qualified assessment rather than relying on a browser result. A useful report should let you see whether the comparison was an upper/lower group, a remaining-item score, or a fitted model, and whether the examinees and administration resemble the situation you care about. Treat a surprising value as a prompt to ask what responses or item features produced it; the number by itself cannot identify a cause or tell you to remove an item. A transparent item report can still help a test developer improve questions or help a reader understand a score, even when it does not establish broad validity. Its value depends on whether its method is clear and the next step stays proportionate to what the evidence can show.

Sources: Test IQ Free 50-question reasoning test