Is the Cognitive Reflection Test a Test of General Intelligence?

What did the original CRT set out to measure?

The original Cognitive Reflection Test (CRT) is a three-item measure designed to capture whether a person checks an immediately appealing response before settling on an answer. Its declared target is this narrow reflection-related performance on problems, not general intelligence as a whole. The raw total is simply the number of the three responses scored correct; by itself, it does not say how a person compares with a population or support an individual IQ interpretation.

In the abstract for *Cognitive Reflection and Decision Making*, the American Economic Association (AEA) describes the CRT as a tool for studying the relation between cognitive ability and decision making. The paper compared responses with decision preferences, including choices involving time, risk, and the status quo, and considered relations with other cognitive measures. This research question matters: the test was introduced to examine whether people who pause over an intuitive answer differ in how they choose, not to supply a comprehensive estimate of intelligence. A three-item total can operationalize performance on that task, but the stated purpose and its correlations do not establish every later use of the score.

The measure is relevant to broader cognition because reflecting on an answer can overlap with capacities that also matter in reasoning and decisions. But relevance is not equivalence. If a score is associated with a cognitive measure or a decision preference, that association describes a relationship among measures in the studied setting. It does not show that the three questions sample the full range of general intellectual ability, or that the raw count has the scale, comparison group, and precision needed to represent an individual's general intelligence. The original CRT therefore offers a compact research indicator of reflection on particular problems. What it measures should be kept separate from what later researchers or test users might hope its score means.

The AEA abstract supplies the original purpose and comparisons, but an abstract-level description cannot establish validity for every form or individual score use. Later research can clarify which skills contribute to performance; it cannot retroactively make the original three-item total a comprehensive intelligence assessment.

Sources: Cognitive Reflection and Decision Making

Is the CRT just a numeracy test?

Numerical CRT performance overlaps with numeracy, but it is not interchangeable with numeracy, and numeracy is not irrelevant. In *Individual Differences in Numeracy and Cognitive Reflection, with Implications for Biases and Fallacies in Probability Judgment*, researchers studied 259 Brazilian undergraduates and 190 American undergraduates. They administered the CRT alongside objective and subjective numeracy measures, and examined how measured dimensions related to probability-judgment outcomes. Numeracy here means understanding and using numbers; the subjective measure captured participants’ self-rated comfort and ability, whereas the objective measure required solving numerical questions.

The Brazilian sample completed tasks involving conjunction and disjunction fallacies; the American sample completed a ratio-bias judgment. Across the studies, the researchers’ factor analyses distinguished CRT performance from numeracy scales and also found that the objective and subjective scales did not measure identical constructs. Their analyses connected different numeracy dimensions to different judgment errors. In the US ratio-bias task, for example, performance was associated with the proportions and relative-magnitude/gist factors, while the CRT-related monitoring factor and the multiplication and verbatim-matching factors were not associated with that outcome. The authors report a broadly similar pattern of differentiated relations across the two samples, rather than one undivided numerical skill predicting every judgment.

That pattern is evidence against calling the CRT merely another numeracy scale: scores grouped and related to outcomes in distinguishable ways within these samples. It is also evidence against saying numeracy contributes nothing. Numerical questions still require a person to make sense of quantities and relations, and some numeracy dimensions were associated with probability judgments. The findings concern measured scales, specific factor solutions, and particular judgment tasks; they do not isolate the mental cause of an answer. A factor or association can show that variables covary in a particular way. It cannot determine whether a given respondent relied on calculation, monitoring, familiarity, a quick surface match, or several processes together. These analyses describe patterns at group level; they do not reveal a private sequence of thought from a correct or incorrect response.

Fresh abstract illustration (not a CRT item or study example): imagine comparing two containers described with different-sized counts. A careful reasoner may translate each description into a common proportion before comparing them. That operation requires understanding quantities, but noticing that a surface count can mislead also involves checking how the problem is framed. The illustration shows why numerical understanding and reflective checking can both matter without being the same ability.

The study’s undergraduate samples and its probability tasks bound the conclusion. The results support treating CRT and numeracy as related but distinct measured dimensions in these settings—not treating a numerical CRT as arithmetic-free, and not generalizing its raw score into a broad ability estimate.

Sources: Individual Differences in Numeracy and Cognitive Reflection, with Implications for Biases and Fallacies in Probability Judgment

What changes when the problems are verbal rather than numerical?

A verbal Cognitive Reflection Test (verbal CRT) can reduce the mathematical content of the task, but it does not isolate reflection from every other ability involved in answering. In *Measuring cognitive reflection without maths: Development and validation of the verbal cognitive reflection test*, the researchers developed a verbal form and retained ten items after development and validation work that included cross-validation. The design addresses a practical limitation of numerical CRT questions: performance on those questions may partly depend on mathematical knowledge or comfort, making it harder to interpret a score when the intended focus is reflective checking.

The study compared the verbal form’s associations with numeracy and math anxiety against those observed for numerical CRT forms. It reported lower associations for the verbal version on these mathematical variables. That pattern is consistent with less mathematical overlap in the verbal task. It matters for measurement because a lower association means that, in the studied samples and forms, participants’ verbal CRT scores tracked numeracy and math anxiety less closely than their numerical CRT scores did. It does not mean those relationships vanished, nor does it show that all mathematical influence was removed. The result supports a relative comparison between versions, not a claim that the verbal score measures reflection alone.

Changing the surface content changes what examinees must understand before reflection can be observed. A verbal problem still requires reading the wording, retaining relevant details, interpreting the question and recognizing which response is tempting but incomplete. Vocabulary, sentence structure, familiarity with the scenario and assumptions about what a question is asking can all affect whether the intended reasoning demand is reached. An item may be non-mathematical and still be easier for people familiar with its language or context. Conversely, ambiguous wording can create difficulty unrelated to reflection. These are design demands, not evidence that the verbal form fails; they are reasons the form needs its own item development and evaluation rather than inheriting conclusions automatically from numerical CRT research.

The comparison therefore answers a bounded question: verbal items showed reduced association with numeracy and math anxiety relative to numerical forms in the reported work. A lower correlation does not, by itself, identify why that reduction occurred or prove that the verbal items captured a purer reflection process. It could reflect reduced calculation demands, differences in wording or content, or other features of the items and samples. The ten-item form offers researchers a way to examine reflection with less overt mathematics, while its score remains a performance measure shaped by comprehension and item design as well as the construct researchers want to study.

Sources: Measuring cognitive reflection without maths: Development and validation of the verbal cognitive reflection test

What can a factor model say about overlap?

A factor model can test whether measured scores are better represented as one shared dimension or as several related dimensions. In *The Factor Structure of Cognitive Reflection, Numeracy, and Fluid Intelligence: The evidence from the Polish adaptation of the Verbal CRT*, researchers examined Polish adaptations of verbal and numerical CRT material alongside numeracy and fluid-intelligence measures. Across their adaptation work, the analysis of a large, diverse Polish adult sample compared candidate structures for the relationships among these scores. The best-fitting tested model separated verbal CRT performance, numeracy and fluid intelligence into three factors; traditional numerical CRT items grouped with numeracy in that model.

This is evidence that the scores did not have to be treated as interchangeable indicators of a single ability in that sample and under the tested model. The placement of traditional numerical CRT items with numeracy is also consistent with the idea that numerical item performance shares meaningful content with numerical understanding. The distinct verbal CRT factor suggests that verbal CRT scores carried a pattern of shared variation not captured by the numeracy factor or the fluid-intelligence factor as specified. In plain terms, people’s relative standing across these measures was structured in a way that a three-part account represented better than the alternatives the researchers tested.

A factor is an inferred summary of covariation: it represents which observed scores tend to vary together, given the measurements and model. It is not a directly observed mental mechanism. When items or scales load together, the result does not establish that one specific process caused each response, that the grouped measures are identical, or that each factor has a single psychological meaning. Nor does a separated factor automatically have validity for every decision someone might want to make from its score. Model fit addresses how well a proposed structure describes the observed relationships; it does not supply a population norm, an IQ conversion, or evidence that an individual score can support a high-stakes interpretation.

The Polish result is therefore useful as a structural finding, not a universal map of cognition. It shows that, among the studied adults and adaptations, the tested measures were better represented by a three-factor arrangement, with numerical CRT items grouped alongside numeracy. Different translations, item sets, populations, administration conditions or statistical specifications could produce a different pattern. The result also cannot establish that verbal CRT is a pure measure of reflection: a separate factor only means its score covaried distinctly in this analysis, not that language comprehension, familiarity or problem-solving demands disappeared. The inference should stay with the evidence: related measures can be distinguishable without being unrelated, and a model-specific separation is not proof that any one score measures broad intelligence.

Sources: The Factor Structure of Cognitive Reflection, Numeracy, and Fluid Intelligence: The evidence from the Polish adaptation of the Verbal CRT

Does the intuitive lure explain what CRT scores predict?

The intuitive lure may be part of what makes a CRT response distinctive, but one study challenges the stronger claim that resisting lures uniquely explains why CRT scores relate to other outcomes. In *A reflection on cognitive reflection – testing convergent/divergent validity of two measures of cognitive reflection*, researchers studied 506 University of Zagreb undergraduates. Their battery included an expanded ten-item numerical CRT, belief-bias syllogisms that also invite intuitive errors, the Berlin Numeracy Test and verbal-reasoning items that do not depend on such lures, a thinking-disposition measure, and decision or heuristics-and-biases tasks. The breadth matters because the analysis could compare lure and non-lure tasks against overlapping outcomes rather than treating CRT performance in isolation.

The authors examined correlations among the measures and latent-factor relationships that account for measurement error, then asked whether the lure tasks related to outcomes more strongly than non-lure measures. In their results, CRT and Berlin Numeracy Test factors were very highly correlated, and CRT and numeracy showed broadly similar relations with thinking disposition and several judgment tasks. Numeracy was more strongly related than CRT to some of those outcomes. The authors therefore argued that the predictive relationships did not arise uniquely from the CRT’s lures: shared abilities and dispositions measured by non-lure tasks could account for much of the same pattern. This is an explanatory challenge to a lure-only account, not a demonstration that reflection never contributes to answering a CRT item.

That analysis asks a different question from *The Factor Structure of Cognitive Reflection, Numeracy, and Fluid Intelligence: The evidence from the Polish adaptation of the Verbal CRT*. The Polish study compared candidate covariance structures in Polish adults and found that a three-factor model best represented verbal CRT, numeracy and fluid intelligence, with traditional numerical CRT items grouped with numeracy. The Croatian study instead examined whether lure versus non-lure measures had distinctive or shared relationships with other tasks and outcomes in its undergraduate sample. A model about which scores covary together is not the same test as an analysis of whether one measure adds a unique explanation for another outcome. The findings are therefore not a direct replication contest, even though both bear on what CRT scores may represent.

Together, the studies support a version- and model-sensitive reading. Numerical CRT items can share substantial measured variance with numeracy; verbal CRT performance can form a distinct factor under a particular adaptation and model; and the predictive pattern in one Croatian student study did not require a unique lure contribution. None of those results alone identifies the process behind each answer or establishes a universal structure across forms, languages, populations and outcomes. A lure can describe an item’s tempting response without proving that lure resistance is the sole source of its associations. Conversely, shared predictive relationships do not erase the possibility that checking an intuitive answer matters in some tasks or analyses. The evidence narrows the claim: CRT prediction is not securely explained by lures alone, while the balance among reflection, numerical skill and other demands remains dependent on the form and the model used.

Sources: A reflection on cognitive reflection – testing convergent/divergent validity of two measures of cognitive reflection; The Factor Structure of Cognitive Reflection, Numeracy, and Fluid Intelligence: The evidence from the Polish adaptation of the Verbal CRT

How do item difficulty and exposure limit a short total?

A three-item total can distinguish some respondents while giving less precise information near the low and high ends of the measured range. In *The Development and Testing of a New Version of the Cognitive Reflection Test Applying Item Response Theory (IRT)*, researchers modeled responses from 438 university students in Italy and the United Kingdom. Item response theory (IRT) estimates how well each question distinguishes performance at different levels of the measured trait; its test information function shows where the full set measures most precisely. In the study’s model, all three original items were above average in difficulty and located toward the positive part of the trait range. Each could distinguish respondents around its own location, but the three-item test provided most information around the middle and substantially less toward the low and high extremes. The authors concluded that the short set could not reliably distinguish lower levels from one another, or higher levels from one another, as well as it did around the center.

This is a limitation of range coverage, not proof that an individual response is meaningless. Someone’s answers still show which of these particular questions they answered correctly under the administration used. But when all items cluster in a relatively difficult region, a low total may compress different levels of lower performance into the same few counts; similarly, near the upper end, the form may not separate stronger performances finely. A raw count has only four possible totals, so even before modeling, it cannot express small differences between examinees. The IRT result adds a more specific point: precision is not uniform across the underlying range, and the study’s university samples and fitted model set the bounds of that finding.

The researchers developed a longer form with additional items at different difficulty levels to extend measurement across a broader range. The logic is practical: items located in different parts of a trait range can contribute information where the original questions contribute less. A longer score is not automatically a measure of general intelligence; it is a response to a coverage limitation for this particular task and research use. Nor does the result mean the original CRT is useless: its items showed discrimination around their locations and may suit questions focused on performance in that region.

The paper also notes that the original items might be familiar to respondents, one reason for developing alternatives. If someone remembers a previously encountered answer, a correct response provides weaker evidence that they independently detected and worked through the tempting response on that occasion. The paper does not establish how many participants had prior exposure or quantify its effect, so familiarity should be recorded when known rather than assumed. Difficulty and exposure thus affect what a short total can resolve: the score describes performance on a small, specific set, with uneven precision across its modeled range and potentially different meaning when an answer is already known.

Sources: The Development and Testing of a New Version of the Cognitive Reflection Test Applying Item Response Theory (IRT)

Does answer format change the result?

A change in response format does not necessarily change the number of correct answers, but similar totals do not make two forms interchangeable for every purpose. *Effect of response format on cognitive reflection: Validating a two- and four-option multiple choice question version of the Cognitive Reflection Test* tested this question in a randomized between-subject experiment with 452 participants aged 18–72. Each participant received one version: open-ended, two-option multiple choice, or four-option multiple choice. Because participants were assigned between groups rather than completing every format, the comparison estimates differences among groups under the study’s administration; it does not show how every person would respond when their own format changes. Here, response format means how an examinee records an answer, with or without supplied alternatives.

The researchers compared correct totals across the three groups and reported no statistically significant difference. That finding qualifies a simple prediction that adding options must raise or lower the total: in this experiment, the observed group totals did not differ enough to meet the study’s significance criterion. It does not establish that the formats are exactly equal. A nonsignificant test can reflect a small effect, uncertainty, or limited ability to detect a difference; a formal equivalence conclusion would require an equivalence design, predefined bounds for a practically negligible difference, and sufficiently precise estimates. The paper’s reported result therefore supports ‘no significant total-score difference detected here,’ not ‘the formats always produce the same score.’

The response pattern contained a separate difference: participants in the two-option version gave more intuitive responses. Correct totals can conceal how answers were reached or which errors became common. Supplied alternatives may change the opportunity to recognize, reject, or select a tempting answer even when the final count of correct responses resembles that in another group. This distinction matters if a researcher uses error type or intuitive responding as the outcome, rather than only the total. The study also reported similar patterns of correlations with other measures and reliability across forms. Those similarities are useful evidence about these versions in this sample, but correlation patterns do not prove identical measurement or authorize every use of the scores.

The multiple-choice versions took less time to complete, according to the study. That is a practical administration difference, not evidence that a faster form measures the same thing with the same precision in all settings. The result is bounded by this item set and experiment. Different translations, rewritten items, time limits, prior exposure, instructions, or delivery conditions could alter both correct totals and response processes. For a narrow comparison using these versions and similar conditions, the study gives reason not to assume that response options necessarily change correct totals. For other forms or uses, comparability remains a question to test rather than a property established by the label ‘CRT.’

Sources: Effect of response format on cognitive reflection: Validating a two- and four-option multiple choice question version of the Cognitive Reflection Test

What evidence would an individual IQ interpretation require?

A defensible individual IQ interpretation needs evidence connecting a person’s observed performance to the proposed score meaning and decision. The *Standards for Educational and Psychological Testing*, jointly issued by AERA, APA, and NCME, frame validity as support for interpretations of scores for their intended uses. In that framework, validity is not a permanent property conferred by a test’s name: evidence must support the interpretation and use being proposed. A score can be informative for one limited research question without supporting a different claim about an individual’s general intelligence.

The inference ladder begins with the response: a person answered particular items correctly or incorrectly under particular conditions. A raw total counts those responses; it does not by itself state how the person compares with a defined population. A norm-referenced interpretation requires an appropriate comparison sample and a documented method for relating an examinee’s performance to that group. Converting a total into an IQ score would require a supported conversion for the exact measure, administration, and population, with enough information to understand the uncertainty around an individual result. The CRT studies discussed here report relationships among task scores, samples, and other measures. Those findings do not supply population norms for individual IQ interpretation or a general conversion from CRT correct totals to IQ.

The distinctions matter because association is a group-level pattern, while an individual interpretation makes a claim about one person. In the Polish adaptation study, for example, the tested model separated verbal CRT, numeracy, and fluid-intelligence factors in its adult sample, with traditional numerical items grouped with numeracy. That covariance structure describes how measured scores related in that analysis. It is not an individual conversion table, a precision estimate for a CRT total, or evidence that the total supports a general-intelligence decision. Likewise, the original paper’s comparisons with decision preferences and cognitive measures made the CRT relevant to broader cognition, but relevance and association do not establish that its score can stand in for a broad intelligence measure.

In general, a broader interpretation would call for measures suited to the intended construct and population, clearly specified and consistent administration, and evidence supporting the proposed score meaning and use. The evidence would need to address how well the measure represents that construct, how precisely it distinguishes people for the intended purpose, and whether the interpretation works for the population and decision at hand. This is a general requirement, not an audit or endorsement of any particular battery. Where the decision has educational, clinical, or other substantial consequences, the interpretation should rest on appropriate assessment and relevant evidence, not a short reasoning task alone.

Rejecting an individual IQ inference does not erase the CRT’s narrower value. Researchers can use it as a defined task variable to study reflection-related performance, and learners can use it as a practice task, provided they describe what was actually measured. The key boundary is the step from observed answers or sample associations to a supported score interpretation: each additional claim requires evidence fitted to that claim. The CRT research summarized here helps characterize task performance and its relationships; it does not establish the norms, precision, conversion, or use-specific support needed to interpret an individual total as general intelligence.

Sources: The Standards for Educational and Psychological Testing; Cognitive Reflection and Decision Making; The Factor Structure of Cognitive Reflection, Numeracy, and Fluid Intelligence: The evidence from the Polish adaptation of the Verbal CRT

How should you describe a result or use practice?

Describe the performance you observed, with enough context that another person can tell what the result covers. A careful report might say: “I answered this version of the Cognitive Reflection Test, in [language], using [open responses or supplied choices], under [timing conditions]. I had [known or uncertain] prior exposure. My result was [number] correct on these items.” Fill in only details you know; if timing or exposure was not recorded, say so instead of guessing. This wording keeps the claim attached to a particular form and occasion. It does not turn a raw count into a population rank, a stable personal trait, or an IQ estimate.

For a low-stakes follow-up, practice the operation that caused hesitation rather than memorizing an answer. For example, when a claim compares two rates, write each as a fraction with the same denominator and check whether the comparison changes. That is an original arithmetic illustration, not a CRT item. Then try a fresh comparison with different quantities and wording. If you can explain why the common denominator helps, you have checked your grasp of that operation on a new example. You have not thereby measured a general change in intelligence, and one familiar exercise cannot establish how performance would carry to unrelated tasks.

Repeated exposure is relevant to how a response should be described: recognizing a previously seen question may reflect memory for its answer as well as current reasoning on that occasion. If you remember an item, note that fact; do not treat a remembered response as a clean demonstration of solving it anew. Likewise, an unfamiliar example can show whether you can apply a specific operation under changed surface details, but it cannot by itself establish broad transfer. Keep the conclusion at the level observed: answers on a specified form, with the stated conditions and known exposure.

What should you do next?

For curiosity or open practice, take the free 50-question test and review its worked answers. Its displayed IQ range is a provisional estimate from an unvalidated conversion with no validated adult reference population; it is not an established comparison with adults. The optional $9 practice report adds explanation, examples and a plan, not validity. Treat it as a learning aid, not evidence for an important decision.

If an educational, clinical or other consequential decision depends on understanding cognitive ability, discuss the question with an appropriately qualified professional. The *Standards for Educational and Psychological Testing* ties validity to evidence supporting a score interpretation and its intended use; the original CRT paper, *Cognitive Reflection and Decision Making*, introduced a brief task for studying reflection and decision making, not an individual IQ conversion. A CRT total alone cannot supply that interpretation. A professional can clarify what question needs answering and what assessment evidence is appropriate to that decision. For a narrow result you simply want to understand, no assessment is needed: describe the form and conditions, and keep the conclusion to the answers given.

Sources: Cognitive Reflection and Decision Making; The Standards for Educational and Psychological Testing