How to Compare Paper and Computer Test Results

How should I compare paper and computer results?

Treat results from paper and computer administrations as separate observations unless evidence for the exact test versions, conditions, population, and intended interpretation supports putting them on a common scale. A higher number on one report is not, by itself, evidence that your cognitive ability increased; a lower one does not establish a decline. Ask whether you are comparing one person over time or score interpretation across a group; evidence for one does not automatically answer the other. A score needs a clear label before it can be compared. A raw score is the number of points earned under the test’s scoring rules. A reported score may instead be transformed using a reference group or a model. Two numbers that look alike might therefore represent different things, while two different numbers might reflect different reporting scales rather than a meaningful difference in performance. Keep labels, versions, dates, and administration details together; the same instrument name does not guarantee the same form, timing, scoring, or interpretation. A correlation between paper and computer scores also has a limited meaning. It describes how closely people’s relative positions correspond; it does not show matching averages, interchangeable scales, or whether one person’s scores can be substituted. In “Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis,” the 1993 synthesis examined 159 corrected cross-mode correlations for timed-power and speeded cognitive ability tests in young-adult and adult populations. It reported an overall correlation of .91, with estimates of .97 for timed-power tests and .72 for speeded tests. This suggests time pressure mattered to cross-mode association in those studies, but the synthesis is not a conversion table for a current test and cannot explain an individual difference. So compare numbers only when the test provider or qualified examiner can point to evidence connecting the exact versions and conditions for the use you have in mind. If no such link is documented, retain each result as a record of performance under its own conditions. Subtraction describes the arithmetic difference, not its cause. Calling it an ability gain or loss requires evidence about score interpretation and individual change. For a low-stakes personal record, note the mode and circumstances and avoid treating the results as a single trend. For a consequential decision, bring both reports and ask a qualified professional whether they support a common interpretation.

Sources: Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis; Guidelines for Technology-Based Assessment

What exactly is being compared: raw performance or a score scale?

First identify whether each number is an earned raw result or a transformed score tied to a reference scale; those are different comparison targets. A raw score is the observed result under a particular administration, such as the number of items answered correctly under that test’s scoring rules. It records performance on that form. It does not, by itself, say how that performance compares with other people, how difficult the items were, or what a difference means. If a paper form yields 34 correct answers and a computer form yields 37, the subtraction is three raw points. It is not automatically a gain in ability: forms may differ in item content, difficulty, timing, or scoring. Preserve each result with its label and administration details before interpreting it.

A scaled score is a transformed result placed on a defined score scale; a norm-referenced score interprets performance against a specified reference group. These labels are not interchangeable. A norm identifies the comparison population and basis, while a scale gives reported values a defined meaning. Online Assessment and the Comparability of Score Meaning describes comparability as commonality of score meaning across testing conditions: interchangeability requires evidence that scores behave similarly across those conditions. Identical raw counts need not imply equal standing if form difficulty differs. Matching reported numbers also do not prove that underlying performances or interpretations match unless the scales were built and checked for that use.

Test developers can use equating or linking to place results from different forms on a common scale for a stated purpose. That bridge needs evidence for the relevant forms and population; a shared test or construct name does not provide it. Paper-based and computer-based admission tests for medicine – are they equivalent? examined two tests, TMS and HAM-Nat, in German applicant samples and found partial rather than full scalar measurement invariance. Some score relationships could be compared under the study’s analysis, but the formats were not shown identical in every relevant respect. The authors note that versions differed, limiting attribution to delivery mode alone. These results are not a conversion rule for another assessment.

Group comparability and individual change answer different questions. Evidence that two modes support comparable group results does not establish that one person’s scores can be subtracted to measure change. That requires relevant evidence about score precision and a method for judging whether an individual difference exceeds expected measurement variation, with attention to practice and changed conditions. Ask what each score represents, whether both belong to a documented common scale, and whether the evidence covers your population and intended decision. Without that bridge, report the results separately as observed performances; do not present their numerical gap as an ability change.

Sources: Online Assessment and the Comparability of Score Meaning; Paper-based and computer-based admission tests for medicine – are they equivalent?

Why can a screen change performance without changing the target ability?

A screen can alter the steps needed to perceive, navigate, answer and manage time, so a difference between paper and computer administrations need not mean that the target ability changed. The examinee must also receive the item, understand its presentation, respond and work within the administration conditions. A score may partly reflect that route through the task. “Making Sense of Mode Effects,” an Ofqual review of research on equivalent paper and digital items, organizes possible causes into four stages of answering a question. It asks where the response process may diverge, without assuming a universal direction. First, an examinee has to perceive the item. A diagram, line, symbol or passage can appear differently according to size, contrast, scrolling, screen dimensions or print layout. Second, the examinee has to interpret what is shown. Line breaks, spacing and visible context can affect how an item is read, even when its content is nominally the same. Third comes interaction: moving through a question, selecting or entering an answer, and managing any on-screen tools. A paper form may permit circling, marking beside a figure, or turning pages; a digital form may provide different ways to select, navigate, flag or revisit items. Interface actions may consume attention or time, especially in timed tests or when several visual details must stay in view. These are possible mechanisms, not evidence that a particular examinee was disadvantaged. Whether it matters depends on the item, interface, familiarity and access needs. Fourth, the examinee must respond under the rules of the session. Timing, the ability to review earlier answers, permitted scratch work, interruptions and device familiarity can differ between administrations. One examinee may be more comfortable annotating paper, another using digital controls. Without evidence on the specific test and users, it is not responsible to predict a universal direction or size for these differences. Record accessibility settings and accommodations because display or input adjustments affect access to the task. The International Test Commission and Association of Test Publishers’ 2025 “Guidelines for Technology-Based Assessment” treat validity, fairness, precision, delivery, scoring and accessibility as connected design and use questions. They are professional guidance, not proof that any one computerized form matches its paper counterpart. Likewise, “Online Assessment and the Comparability of Score Meaning” frames comparability around whether scores support the same interpretation, not whether two screens or pages look alike. Alongside Ofqual’s framework, this suggests a practical record: exact form and mode, device or paper format, timing and navigation rules, scratch-work and review permissions, accessibility settings, setting and interruptions. Similar-looking items alone do not establish identical task experience. Record these conditions before comparing results; they help identify plausible explanations, while test-specific evidence is still needed to decide whether scores share an interpretation.

Sources: Making Sense of Mode Effects; Guidelines for Technology-Based Assessment; Online Assessment and the Comparability of Score Meaning

What does cognitive-test research say about mode equivalence?

The research does not justify a universal rule that paper and computer results are interchangeable. A close relationship between people’s scores in two formats can coexist with a difference in average scores, and neither finding alone tells an individual whether a change in their own result reflects changed ability. “Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis” pooled 159 cross-mode correlations from studies of young adults and adults: 123 correlations came from timed-power tests, where time is allowed but speed is not the central constraint, and 36 from speeded tests, where completing items quickly is part of performance. After correcting estimates for measurement error, the reported overall correlation was .91; estimates were .97 for timed-power tests and .72 for speeded tests. These correlations describe how strongly people’s relative standing in one mode tracked their relative standing in the other across the included studies. They do not show that the two modes produced identical average scores, shared a common score scale, or would yield the same result for every person. It predates current devices and interfaces. Its pooled association cannot supply a conversion for an unrelated test or establish comparability for a particular modern administration. A later example shows why conclusions should stay tied to the instrument and study design. “Paper-based and computer-based admission tests for medicine – are they equivalent?” examined Germany’s Test for Medical Studies (TMS) and Hamburg Assessment Test for Medicine, Natural Sciences (HAM-Nat), used in university selection. In a large, representative sample of German university applicants, the researchers tested whether the measures behaved consistently across paper and computer formats using multiple-group confirmatory factor analysis. They reported partial scalar invariance for both tests, describing this as initial evidence of equivalence across formats. Partial invariance does not declare every item or score identical. Its tests, applicant population, format transition and statistical model define the scope of the finding. The authors note that earlier work using more advanced equivalence methods was limited and varied in design, method and ability measured, making results difficult to compare; they recommend investigating equivalence case by case. Thus, these two German admission tests showed initial, partial evidence under the study’s analysis. It cannot establish what happens with a different reasoning test, a different population, another screen or an individual retaking a test years later. Taken together, the studies answer different questions. The meta-analysis summarizes associations in relative ordering across many older comparisons; the admissions study asks whether specified measures function comparably across formats in one high-stakes selection context. Neither licenses a generic paper-to-screen adjustment. Before treating results as one scale, ask whether the exact forms and target population were studied, whether the evidence addresses score comparability rather than correlation alone, and whether the intended inference is about group averages or one person’s change. Without that bridge, keep the two results as separate performances under different conditions.

Sources: Equivalence of computerized and paper-and-pencil cognitive ability tests: A meta-analysis; Paper-based and computer-based admission tests for medicine – are they equivalent?

How do calibration and score linking make comparison defensible?

Comparable interpretation needs an evidence bridge built for the exact test versions, modes, population, and intended use. A shared name or similar-looking questions cannot supply it. Score linking transforms results onto a scale for a defined comparison. Equating is a more demanding form intended to make scores interchangeable; other links may support a narrower comparison, such as predicting one score from another. The International Test Commission’s July 2025 Guidelines for Technology-Based Assessment caution that comparability and interchangeability are not synonyms. A link adequate for broad categories may not justify treating two individual scores as identical measures.

Developers can investigate the bridge by administering both modes under controlled conditions to comparable groups, or by using common items or examinees to connect forms. They examine whether items behave similarly across modes and whether score distributions, reliability, and measurement error support the proposed interpretation. Measurement invariance means that the measurement relationship is sufficiently stable across the groups or modes being compared; item-level analyses can reveal where it fails. A conversion table alone does not show that these checks were done or that a link applies to another age group, device, form, or decision. The 2025 Guidelines call for evidence addressing relevant technologies, administrative conditions, forms, item differences, score distributions, reliability, and standard error. Developers must support comparability claims; users must avoid introducing unstudied differences during administration.

PISA 2022 illustrates both the value and limits of this work. The OECD describes a randomized mode-effect study in the 2015 field trial, in which students in equivalent international samples answered common items on paper or computer. Most items supported comparability, but some showed different difficulty, and sometimes different relationships with proficiency, in the computer mode. PISA therefore used a partial-invariance approach: it accounted for mode-sensitive items while linking results with common items that had stable characteristics. The OECD reports that many shared items supported a strong connection between the scales and mean comparisons across countries using different modes. Yet it also warns that many computer-based reading items, and some science items, were absent from paper forms; conclusions about paper scores based on the full computer item set therefore require caution. Its paper mathematics results also needed interpretation against earlier PISA proficiency levels, not the newer computer assessment’s levels.

That example does not provide a conversion for another reasoning test. It shows what a defensible comparison requires: evidence about the particular items and forms, a stated target population, a linking method, and limits on the conclusions. Before combining two results, look for documentation naming both versions and modes, describing the study design and score transformation, and reporting item functioning and score precision for the relevant population. Without that bridge, preserve results as separate observations. A published conversion applies only within the evidence and population that support it.

Sources: How comparable are the PISA 2022 computer- and paper-based tests?; Guidelines for Technology-Based Assessment; Guidelines for Technology-Based Assessment

What do ACT and device studies demonstrate—and not demonstrate?

ACT shows why a comparison must be built for a particular test program. In Three Studies of Comparability Between Paper-Based and Computer-Based Testing for the ACT, the publisher reports three randomized equivalent-group studies conducted on national test dates in October 2019, December 2019 and February 2020. Students took the same ACT form in one of two modes, on paper or online, using the TAO platform on non-convertible Chromebooks. Before adjustment, average online scores were higher in all three studies for English and reading: 0.10–0.13 standard deviations for English and 0.16–0.22 for reading. Math differences were smaller, from a nonsignificant 0.04 to 0.12 in science. The report found broadly similar relationships among subject scores across modes. Similar measurement therefore did not mean identical observed scores under each mode.

ACT then used random-group equating to create mode-adjusted score conversions for its own program. Equating places forms or conditions on a common scale for a defined use. After adjustment, average online scale scores were nearly identical to paper averages in the randomized groups. This shows that a testing organization can study mode effects and make a test-specific adjustment; it is not a correction factor for another assessment. The research concerns ACT items, students, platform, administration and scoring. The report says future mode changes may warrant further study. A reader cannot subtract an ACT-like average from a personal score: group-level adjustment does not estimate one person’s ability change.

The Mayo Clinic Study of Aging’s Cogstate PC/iPad comparison illustrates a different boundary. Cognitively unimpaired adults aged 50 and older completed the Cogstate Brief Battery and Groton Maze Learning Test in clinic. Three cohorts were examined: 168 adults tested on both devices the same day; baseline results from 1,820 PC users and 605 iPad users; and matched groups of 63 per device followed over three visits across 15 months. Several Cogstate measures showed small but statistically significant faster performance on PC; accuracy usually did not differ or differed only slightly. The maze task showed faster performance and more total errors on iPad. Most variables showed no difference in rate of change between devices. The authors say further research is needed to know whether differences affect clinical interpretation or research outcomes.

This study compares two computer devices, not paper with computer, and a specific neuropsychological battery, not general intelligence tests. It supports a narrow lesson: even a device change can matter differently by task and outcome. Together, these cases give no universal direction, size or correction for mode effects. They show why evidence must match the instrument, versions, population, conditions and intended interpretation. Treat two results as separate observations unless the provider supplies applicable comparison evidence. Use these studies as examples of how evidence can be built, never as a numerical adjustment for your test.

Sources: Three Studies of Comparability Between Paper-Based and Computer-Based Testing for the ACT; PC and iPad modality comparison in the Cogstate Brief Battery

How should I interpret a difference across two administrations?

A difference between two administrations does not by itself show that ability changed. It is an observed difference between scores produced on particular occasions; attributing it to a change in the person requires accounting for other explanations. Subtraction describes the arithmetic, but cannot identify its cause. A difference might reflect change in the ability measured, a change in testing conditions, familiarity with the task, or measurement imprecision. These explanations can also operate together. A score is an estimate based on a limited set of responses, not a direct reading of a fixed quantity. The standard error of measurement describes expected uncertainty around a score under a particular test’s model and conditions. It belongs to that instrument and scale; it cannot be borrowed from a different test or inferred from the size of the difference. If the publisher supplies precision information, use it to understand uncertainty in each score, not as proof that two administrations show change. *Online Assessment and the Comparability of Score Meaning* explains that scores from different conditions require comparable meaning before they can be treated as interchangeable. Retaking a test introduces another possible influence. Remembering its format, understanding instructions more quickly, or seeing familiar material can affect later performance. That does not mean every higher retest score is a practice effect, or that practice has changed general intelligence. Consider familiarity alongside the interval between sessions and any preparation. The International Test Commission and Association of Test Publishers’ *Guidelines for Technology-Based Assessment* treats score precision, comparability, and measuring change as distinct technical questions. The guidance supplies no formula for correcting an individual result; interpretation requires evidence suited to the assessment and intended use. Conditions may also differ: time limits, pacing rules, device or display, interruptions, setting, permitted aids, accommodations, health, or sleep. Record what is known rather than guessing how much any factor mattered. *Paper-based and computer-based admission tests for medicine – are they equivalent?* studied two German medical-admission tests in applicant samples. It reported partial scalar invariance for both, while full scalar invariance was not consistently supported in validation samples. This group-level evidence concerns those tests and formats; it cannot explain an individual’s difference or establish a reliable-change threshold for another instrument. For a personal comparison, ask the publisher or qualified examiner whether a reliable-change method applies to the exact instrument, versions, interval, and conditions. It should account for test precision and the comparison’s purpose. Without such guidance, do not call a difference a gain, decline, or ability change. Keep the scores as separate observations with their dates and administration details. For consequential decisions, bring both reports to a qualified professional for interpretation.

Sources: Online Assessment and the Comparability of Score Meaning; Guidelines for Technology-Based Assessment; Paper-based and computer-based admission tests for medicine – are they equivalent?

What should I record and ask before using the result?

Record each administration’s exact test and version or form, date, mode, device and input method, time limit, setting, interruptions, accommodations, and displayed score labels. Keep the raw number correct separate from any scaled or norm-referenced score, and save each report. These details help identify whether a difference might also reflect a different form, device, timing rule, or condition. Ask the publisher or examiner: Were these exact forms and modes studied together, and in which population? Was a linking or conversion method established for this comparison and intended use? Does the reported number represent raw performance or a shared scale? What precision information applies, and is there a method for judging whether an individual change exceeds expected measurement error? The International Test Commission and Association of Test Publishers’ 2025 Guidelines for Technology-Based Assessment cover design, delivery, scoring, fairness, accessibility, security, and privacy. They guide these questions; they do not prove that a particular pair of scores is comparable. The ACT comparability report shows why the specific program matters: its randomized studies examined ACT modes, and ACT used adjusted conversions for comparability. That does not provide a conversion for another test. If education, clinical care, or another consequential decision depends on the result, take both reports and your condition notes to a qualified professional. Ask for an interpretation tied to the instrument’s evidence and intended use.

Sources: Guidelines for Technology-Based Assessment; Three Studies of Comparability Between Paper-Based and Computer-Based Testing for the ACT

What is a responsible next step?

For low-stakes practice, choose one format and keep conditions consistent. Record the test and version, date, mode, device, timing, setting, and accommodations. If an attempt differs, note what changed. This cannot make scores equivalent, but helps prevent a difference being mistaken for ability change. For consequential decisions, ask the provider or a qualified examiner whether evidence links those versions and conditions, which scale applies, and how uncertain the result is. The 2025 International Test Commission and Association of Test Publishers Guidelines for Technology-Based Assessment address technology-based assessment practice; guidelines alone do not validate a particular comparison.

Test IQ Free offers a free 50-question reasoning test and worked answers for practice. Its estimated IQ range comes from an unvalidated conversion with no validated adult reference population. It is not normed, a confidence interval, or an established comparison with other adults. Review your reasoning; do not treat the range as identity or fixed potential. The optional $9 report adds explanation and a practice plan, but uses the same provisional conversion and does not add validation. For an informal baseline, repeat under comparable conditions and describe any difference as observed performance unless evidence for that test supports a stronger interpretation.

Questions readers ask