How the SHSAT Questions Are Analyzed

The counts below cover the 723 questions analyzed so far and their 2171 classified wrong answers.

How many of these questions are new, and how many the DOE has printed before

How each question is analyzed

Every released SHSAT question on this site carries the same structured analysis, so questions can be compared and patterns can be counted across the whole exam. It answers four questions: what the question tests (the New York State standards and the grade they belong to), how much thinking it demands, what makes it hard, and what each wrong answer reveals about the student who chooses it.

The categories below come from research on why test questions are hard (item difficulty modeling) and on what wrong answers tell us (distractor analysis and cognitive diagnosis). Each category lists its sources.

Difficulty: estimated and measured

In testing, an item’s difficulty is the share of students who answer it correctly, its p-value: a question 80% of students get right is easy, and one 30% get right is hard. Each question here has an estimated difficulty from 1 to 5, judged from its content before anyone answered it. Once enough students have answered a question on this site, its page also shows the share who got it right in the last 30 days.

The two can differ. The measured figure depends on who is practicing, and students on this site are not a random sample of SHSAT takers. A question that turns out much harder than estimated is a sign of a trap the estimate missed.

1: Routine: one familiar step, little to misread.
2: Straightforward: two steps or one careful reading.
3: Moderate: several steps, or a close reading with one strong distractor.
4: Hard: combines ideas, needs a plan, and has more than one tempting wrong answer.
5: Among the hardest on the SHSAT: a non-obvious approach, heavy reading, or a well-hidden trap.

What makes a question hard

Research that predicts item difficulty from item features finds that difficulty comes from identifiable sources: how much there is to read, how many steps the solution takes, how close the wrong answers are to the right one, and so on (Embretson & Wetzel, 1987; Enright, Morley & Sheehan, 2002; Gorin & Embretson, 2006). Each question lists its main difficulty drivers, the biggest first, sorted into three phases of solving it: understanding the question, working it out, and choosing the answer. Each driver is rated minor, moderate or major for that question.

Some drivers matter for fairness as well as difficulty. In a math question, long sentences and hard words make the question harder without testing more math. Testing experts call this construct-irrelevant difficulty (Messick, 1989; Abedi & Lord, 2001), and recognizing it helps a student see that the math itself may be easier than it looks.

Understanding the question

Reading the text, the question and any figure, and building a correct picture of the task.

A lot to read: A long or dense text, or a long question, with many separate facts to take in. (Kintsch, 1998; Gorin & Embretson, 2006; Freedle & Kostin, 1993) 23 questions
Complex sentences: Long sentences, clauses inside clauses, passive voice, or literary word order, in the text or in the question. In a math question this is reading difficulty, not math difficulty. (Abedi & Lord, 2001; Freedle & Kostin, 1993; Messick, 1989) 64 questions
Hard words: Uncommon or academic words whose meaning matters, in the text, the question or the choices. (Perfetti, 2007; Freedle & Kostin, 1993) 62 questions
A question in the negative: The question asks what is NOT true, the EXCEPTION or the LEAST, which reverses the usual way of checking choices. (Haladyna, Downing & Rodriguez, 2002) 1 question
An unfamiliar subject: The text is about something few students know about, such as a scientific process or a historical event, so the text must supply all of the meaning. (Recht & Leslie, 1988; Kintsch, 1998) 3 questions
Figurative or literary language: Metaphor, irony, tone, imagery or poetic form: the meaning is not on the surface of the words. (Graesser, Singer & Trabasso, 1994; Kintsch, 1998) 68 questions
Turning words into math: The situation has to be turned into an equation, expression or diagram. Relational wording such as "more than", "times as many" or "of" invites a word-by-word translation that is wrong. (Hegarty, Mayer & Monk, 1995; Koedinger & Nathan, 2004) 123 questions
Reading a figure, table or graph: Key information is in a figure, number line, table, graph or map, or has to be carried from one representation to another. (Lesh, Post & Behr, 1987; Kirsch & Mosenthal, 1990) 80 questions
Variables instead of numbers: The problem works with letters, general cases or unknown quantities rather than concrete numbers, so there is no number to check against. (Koedinger & Nathan, 2004; Enright, Morley & Sheehan, 2002) 36 questions

Working it out

Choosing a method and carrying it out, or finding and connecting the evidence.

Many steps: A long chain of operations or sub-goals. Each step is another chance for an error, and errors carry forward. (Daniel & Embretson, 2010; Enright, Morley & Sheehan, 2002) 67 questions
Much to keep track of: Several values, conditions, characters or ideas have to be held in mind at once. (Sweller, 1988; Carpenter, Just & Shell, 1990) 65 questions
No obvious method: The wording does not signal the method. The problem needs a plan or a less common approach: working backward, testing cases, or seeing a hidden structure. (Schoenfeld, 1985) 18 questions
Combines ideas: Two or more concepts or skills have to be combined in one solution, such as ratio with area, or tone with structure. (Embretson, 1998; Daniel & Embretson, 2010) 28 questions
Error-prone arithmetic: Fractions, negative numbers, decimals, exponents or large numbers, where a careless slip is likely even with the right method. (Brown & Burton, 1978; Junker & Sijtsma, 2001) 130 questions
Something you must already know: A formula, fact, rule or word meaning that the test does not give. (Anderson & Krathwohl, 2001) 110 questions
Reading between the lines: The answer is not stated. It has to be inferred from what is stated: a motive, a purpose, an implication or a connection between sentences. (Graesser, Singer & Trabasso, 1994; Kintsch, 1998) 164 questions
Evidence in several places: The evidence is spread across the text or far from where the question points, so it has to be found and put together. (Kirsch & Mosenthal, 1990; Gorin & Embretson, 2006) 107 questions
How, not what: The question asks why the author wrote something, or how a part works, such as its structure, purpose or effect, rather than what the text says. (Anderson & Krathwohl, 2001) 85 questions
A rule that doesn't sound wrong: The error or its fix depends on a grammar, usage or punctuation rule that many students don't hear as wrong, such as a dangling modifier or a comma splice. (Connors & Lunsford, 1988) 16 questions
How sentences fit together: Placing, adding, joining or connecting sentences means tracking the paragraph's line of thought: its transitions, references and focus. (Halliday & Hasan, 1976) 35 questions

Choosing the answer

Comparing the choices with what you found and ruling out the wrong ones.

A very close wrong answer: At least one wrong choice is so like the right one that only a careful check tells them apart. In reading, it matches the right choice except for one word, one part of a claim or one detail of the text. In math, it is numerically close to the right answer or written almost the same way, such as x + 1/2 and (x + 1)/2, so estimating cannot rule it out. A wrong answer that is simply a common mistake, such as a value from partway through, is classified with the wrong answers, not here. (Embretson & Wetzel, 1987; Gorin & Embretson, 2006; Ascalon et al., 2007) 229 questions
The right answer is reworded: The right choice paraphrases the text in new words, while wrong choices repeat the text's own words and so look better supported. (Freedle & Kostin, 1993; Kirsch & Mosenthal, 1990) 48 questions
Every choice must be checked: The choices can't be ruled out at a glance. Each has to be tested against the text or worked out, as in "which of the following is true". (Embretson & Wetzel, 1987) 117 questions
An answer in an unexpected form: The result has to be recognized when written another way, or, for a grid-in, there are no choices to check against. (Enright, Morley & Sheehan, 2002) 57 questions
Slow to do carefully: A reliable method takes well over the average of about a minute and a half per question, which tempts students to take a shortcut or guess. (Lu & Sireci, 2007)

Depth of knowledge and thinking process

Two standard scales describe the kind of thinking a question needs. Webb’s Depth of Knowledge (Webb, 1997) rates the complexity of the task: 1, recall and reproduction; 2, skills and concepts; 3, strategic thinking. Level 4, extended thinking, needs days of work and does not occur on a timed test. The revised Bloom’s taxonomy (Anderson & Krathwohl, 2001) names the main thinking process: remember, understand, apply, analyze or evaluate.

Depth of knowledge

1, recall and reproduction: Recall a fact, definition, formula or rule, or carry out a routine one-step procedure. The answer is either known or quickly found. (Webb, 1997) 122 questions 17%
2, skills and concepts: Make a decision about how to approach the task: apply a concept, carry out a procedure of two or more steps, compare, classify, or summarize what the text says. (Webb, 1997) 511 questions 71%
3, strategic thinking: Reason, plan and use evidence: the path is not routine, there is more than one possible approach, or the answer must be justified by drawing on the text as a whole. (Webb, 1997) 90 questions 12%

Thinking process

Remember: Retrieve knowledge from memory: a fact, formula, rule or word meaning. (Anderson & Krathwohl, 2001) 1 question 0%
Understand: Build meaning: interpret, paraphrase, summarize, explain, or infer what is stated. (Anderson & Krathwohl, 2001) 84 questions 12%
Apply: Carry out or use a procedure in a given situation. (Anderson & Krathwohl, 2001) 330 questions 46%
Analyze: Break material into parts and work out how they relate: structure, purpose, the author's point of view, or which facts matter. (Anderson & Krathwohl, 2001) 213 questions 29%
Evaluate: Judge against criteria: which revision is most effective, which evidence best supports a claim. (Anderson & Krathwohl, 2001) 95 questions 13%

Wrong answers are data

The wrong choices on a well-written test are not random. Each is built to attract students who make a particular mistake, which is why test designers call them distractors (Gierl et al., 2017). Every wrong answer here is classified three ways:

Type: the mistake that leads to it, such as stopping a step early or choosing a detail when the question asks for the central idea.
What it reveals: a misconception (a consistent wrong belief), partial knowledge (on the right track but incomplete), a slip (a careless error by someone who knows how), or a surface strategy (matching numbers or words without understanding). This follows research that models students’ answers to diagnose what they know (Tatsuoka, 1983; Junker & Sijtsma, 2001).
Expected pull: how many students it is likely to attract. Wrong answers drawn by partial knowledge pull students who know a good deal (Thissen, Steinberg & Fitzpatrick, 1989). A wrong answer picked by fewer than about 5% of students is considered non-functioning (Haladyna & Downing, 1993).

Once a question has enough answers on this site, its page shows how the answers actually split between the choices, next to the expected pull.

What a wrong answer reveals

A misconception: A consistent wrong belief or rule, such as "percents add" or "a bigger denominator means a bigger fraction". A student who holds it makes the same error on other questions, so it is worth unlearning. (Tatsuoka, 1983; Smith, diSessa & Roschelle, 1993; Brown & Burton, 1978) 401 wrong answers, 18%
Partial knowledge: On the right track but incomplete: stops a step early, gets one half right, or misses one condition. These wrong answers attract students who know a good deal, which makes them the hardest to resist. (Thissen, Steinberg & Fitzpatrick, 1989) 1023 wrong answers, 47%
A slip: The student knows how but makes a careless error: a misread word or number, or an arithmetic slip. Checking work is the cure. (Junker & Sijtsma, 2001) 201 wrong answers, 9%
A surface strategy: No real understanding of the problem: combining the numbers in the question, matching words from the text, or picking what looks familiar. These wrong answers attract students who are guessing. (Hegarty, Mayer & Monk, 1995; Freedle & Kostin, 1993) 546 wrong answers, 25%

Expected pull

Strong: Expected to be the wrong answer most students choose, including some who know the material well. 621 wrong answers, 29%
Moderate: Expected to draw a fair share of students who are unsure. 775 wrong answers, 36%
Weak: Expected to draw few students. A wrong answer chosen by fewer than about 5% of students does little work, and test designers count it as non-functioning. (Haladyna & Downing, 1993; Rodriguez, 2005) 775 wrong answers, 36%

Traps and wrong-answer types: math

Stops one step early: A choice is a number from partway through the solution. (King et al., 2004; Gierl et al., 2017) 105 questions
Answers a different question: A choice is right for another quantity: the other unknown, the rest instead of the part, the part instead of the whole. (King et al., 2004; Hegarty, Mayer & Monk, 1995) 87 questions
Percent of the wrong amount: Finding the whole from a part, or a percent change measured from the wrong starting amount. (Parker & Leinhardt, 1995) 11 questions
Units or scale: The answer is wanted in other units, or a rate or scale must be converted first. (King et al., 2004) 16 questions
Counting the ends: Inclusive or exclusive ends, fence posts, consecutive numbers: an answer that is one too many or too few, or an endpoint drawn open where it belongs closed (or the reverse). (Brown & Burton, 1978) 25 questions
Negatives and direction: Arithmetic with negative numbers, the order of a subtraction, or direction on a number line. (Moving a term across the equals sign without changing its sign is a faulty procedure.) (Brown & Burton, 1978) 50 questions
Order of operations: Working left to right, or distributing a minus sign wrongly, gives a choice. (Brown & Burton, 1978) 13 questions
Trusting the picture: The figure suggests something it does not state; work from the given facts. (Lesh, Post & Behr, 1987) 4 questions
Information you don't need: The question gives numbers or facts that play no part in the answer. (Hegarty, Mayer & Monk, 1995) 8 questions
Overlapping groups: Things in two groups at once are counted twice, or not subtracted. (King et al., 2004) 8 questions
A formula you must know: Needs a formula the test does not give: area, circumference, volume, probability rules. (Anderson & Krathwohl, 2001) 15 questions
Answer in another form: The right answer is written differently from the way you worked it out. (Enright, Morley & Sheehan, 2002) 8 questions
Without replacement: In a second pick there is one fewer item, which changes the probability. (Fischbein & Schnarch, 1997) 9 questions
Wrong operation: Adds instead of multiplying, divides the wrong way round, turns a ratio or rate upside down, or undoes a step in the wrong direction (subtracts what should be added back). (Hegarty, Mayer & Monk, 1995; King et al., 2004) 68 questions
Follows a key word: Picks the operation from a single word rather than the situation: "removed" so subtract, "of" so multiply, "more" so add, when the situation needs something else. (Hegarty, Mayer & Monk, 1995) 6 questions
Equation written backwards: Writes a comparison the wrong way round, word for word: "six times as many students as professors" as 6S = P. The bigger quantity gets the bigger number. (Clement, Lochhead & Monk, 1981) 4 questions
Checks only one number: Tries one value in each choice and takes the first that works, when that value happens to work for a wrong choice too; a second value would rule it out. (Schoenfeld, 1985) 6 questions
Rounds too early: Rounds a value partway through, or rounds the wrong way, so the final answer drifts from the exact one. (King et al., 2004) 13 questions
Wrong formula: Uses a related but wrong formula: perimeter for area, diameter for radius, area for volume, or a misremembered one. (King et al., 2004) 23 questions
Numbers taken from the question: Combines the numbers in the question without a model of the situation, or is simply a number that appears in the question. (Hegarty, Mayer & Monk, 1995) 89 questions
A faulty procedure: A systematic but wrong procedure: adding fractions straight across, applying an exponent to each term of a sum, dropping a sign when moving a term. (Brown & Burton, 1978; Tatsuoka, 1983) 53 questions
Misunderstands the concept: A wrong belief about a concept: that percents can be added, that area grows like length, that averages can be averaged, that a bigger denominator means a bigger fraction. (Smith, diSessa & Roschelle, 1993; De Bock et al., 1998; Ni & Zhou, 2005) 93 questions
An arithmetic slip: The right method with a plausible arithmetic error: a digit, a place value or a times-table fact. (Junker & Sijtsma, 2001) 35 questions
Misreads the question: Misses or misreads a condition or a word in the question: "at least", "each", "remaining", "not". (Hegarty, Mayer & Monk, 1995) 76 questions
Misreads the figure: Reads a value, label or tick spacing in a figure, table or graph wrongly. (Lesh, Post & Behr, 1987; Kirsch & Mosenthal, 1990) 21 questions

Traps and wrong-answer types: reading

True but not the answer: A choice says something accurate about the text that does not answer this question. (King et al., 2004; Embretson & Wetzel, 1987) 57 questions
Too narrow: A single detail offered as the central idea or a summary, or a summary that keeps minor details and leaves out an essential part. (King et al., 2004) 52 questions
Too broad or too strong: A choice overstates, generalizes, or uses absolute words the text does not support. (King et al., 2004) 53 questions
Wrong part of the text: A choice is true of another paragraph, stanza or speaker than the one asked about. (Kirsch & Mosenthal, 1990) 87 questions
Outside knowledge: A choice is plausible in the real world but not supported by this text. (King et al., 2004; Recht & Leslie, 1988) 49 questions
Reversed relationship: Cause and effect, or the order of events, turned around. (King et al., 2004) 10 questions
Half right: One half of a two-part choice is right and the other is not. (King et al., 2004; Thissen, Steinberg & Fitzpatrick, 1989) 49 questions
Taking figurative language literally: A choice reads a figure of speech or image at face value. (Graesser, Singer & Trabasso, 1994) 17 questions
Dictionary meaning, not meaning here: A common meaning of the word that does not fit its use in this text. (Perfetti, 2007) 13 questions
Supports only part of the claim: In a question about evidence: a line that backs one half of the claim but not the other, or a related point rather than the claim itself. (King et al., 2004; Thissen, Steinberg & Fitzpatrick, 1989) 56 questions
Twists what the text says: Uses the text's topic and ideas but changes what it says: who did it, how much, when, or why. (King et al., 2004) 128 questions
Word match: Repeats striking words or phrases from the text, so it looks supported, but its meaning does not fit the question. (Freedle & Kostin, 1993) 23 questions
Wrong purpose or structure: Names a purpose, structure or effect that the part does not serve, or describes what a part says when the question asks what it does. (Anderson & Krathwohl, 2001) 45 questions
Reads too much in: A reasonable-sounding conclusion that goes further than the text allows. (Graesser, Singer & Trabasso, 1994) 137 questions
Wrong tone or attitude: Misjudges the tone, attitude or point of view, such as reading admiration as criticism, or taking a character's or subject's feelings for the author's. (Graesser, Singer & Trabasso, 1994) 53 questions

Traps and wrong-answer types: revising and editing

Fixes one error, adds another: A revision corrects the problem but creates or keeps another error. (Connors & Lunsford, 1988)
Changes the meaning: A revision is correct English but no longer says what the writer meant. (Connors & Lunsford, 1988) 6 questions
Wordy, vague or informal: A revision is grammatical but wordy, imprecise or too casual for the passage. (Connors & Lunsford, 1988) 13 questions
Fragment or run-on: Creates or keeps a sentence fragment, a run-on sentence or a comma splice. (Connors & Lunsford, 1988) 1 question
Agreement: A verb that does not agree with its subject, or a pronoun that does not agree with the word it stands for. (Connors & Lunsford, 1988) 2 questions
Misplaced modifier: A describing word or phrase that attaches to the wrong word, or to nothing in the sentence. (Connors & Lunsford, 1988) 2 questions
Verb tense or form: The wrong tense, an inconsistent shift of tense, or a wrong verb form. (Connors & Lunsford, 1988) 3 questions
Unclear pronoun: A pronoun whose reference is vague or ambiguous, or one in the wrong case. (Connors & Lunsford, 1988) 1 question
Punctuation: A comma, semicolon, colon, dash or apostrophe used where it does not belong, or missing where it does. (Connors & Lunsford, 1988) 8 questions
Not parallel: Items in a list or comparison are not in the same grammatical form. (Connors & Lunsford, 1988)
Wrong transition: A connecting word or phrase that signals the wrong relationship, such as contrast where the idea continues. (Halliday & Hasan, 1976) 11 questions
Off the point: Adds or keeps something that does not support the paragraph's or the passage's focus, or an introduction or conclusion that misses the main point. (Halliday & Hasan, 1976) 17 questions
Says it twice: Repeats an idea that the sentence or passage already states. (Connors & Lunsford, 1988) 3 questions
Sentence in the wrong place: Puts a sentence where it breaks the order of ideas or separates it from what it refers to. (Halliday & Hasan, 1976) 5 questions
Fixes what isn't broken: Changes a part of the sentence that was already correct, while the real error stays; or, where the question asks which sentence has an error or should go, picks one that is correct and belongs. (Connors & Lunsford, 1988) 16 questions

Traps and wrong-answer types: either section

NOT, LEAST or EXCEPT: The question asks for the exception, the least, or what is not true. (Haladyna, Downing & Rodriguez, 2002) 2 questions
Little reason to choose it: No common line of thinking leads here. Few students are expected to pick it, so it mostly catches guesses. (Haladyna & Downing, 1993; Rodriguez, 2005) 49 questions

What this analysis cannot tell you

The analysis is written by reviewers who read each question, its passage and the DOE’s explanation; it is an informed judgment, not a measurement. The answer counts are anonymous and cover only the first answer each browser gives to a question, so they cannot show how well a question separates strong students from weak ones (its discrimination), which would need each student’s overall score.

References

Abedi, J., & Lord, C. (2001). The language factor in mathematics tests. Applied Measurement in Education, 14(3), 219–234.
Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. Longman.
Ascalon, M. E., Meyers, L. S., Davis, B. W., & Smits, N. (2007). Distractor similarity and item-stem structure: Effects on item difficulty. Applied Measurement in Education, 20(2), 153–170.
Brown, J. S., & Burton, R. R. (1978). Diagnostic models for procedural bugs in basic mathematical skills. Cognitive Science, 2(2), 155–192.
Carpenter, P. A., Just, M. A., & Shell, P. (1990). What one intelligence test measures: A theoretical account of the processing in the Raven Progressive Matrices Test. Psychological Review, 97(3), 404–431.
Clement, J., Lochhead, J., & Monk, G. S. (1981). Translation difficulties in learning mathematics. The American Mathematical Monthly, 88(4), 286–290.
Connors, R. J., & Lunsford, A. A. (1988). Frequency of formal errors in current college writing, or Ma and Pa Kettle do research. College Composition and Communication, 39(4), 395–409.
Daniel, R. C., & Embretson, S. E. (2010). Designing cognitive complexity in mathematical problem-solving items. Applied Psychological Measurement, 34(5), 348–364.
De Bock, D., Verschaffel, L., & Janssens, D. (1998). The predominance of the linear model in secondary school students' solutions of word problems involving length and area of similar plane figures. Educational Studies in Mathematics, 35(1), 65–83.
Embretson, S. E. (1998). A cognitive design system approach to generating valid tests: Application to abstract reasoning. Psychological Methods, 3(3), 380–396.
Embretson, S. E., & Wetzel, C. D. (1987). Component latent trait models for paragraph comprehension tests. Applied Psychological Measurement, 11(2), 175–193.
Enright, M. K., Morley, M., & Sheehan, K. M. (2002). Items by design: The impact of systematic feature variation on item statistical characteristics. Applied Measurement in Education, 15(1), 49–74.
Fischbein, E., & Schnarch, D. (1997). The evolution with age of probabilistic, intuitively based misconceptions. Journal for Research in Mathematics Education, 28(1), 96–105.
Freedle, R., & Kostin, I. (1993). The prediction of TOEFL reading item difficulty: Implications for construct validity. Language Testing, 10(2), 133–170.
Gierl, M. J., Bulut, O., Guo, Q., & Zhang, X. (2017). Developing, analyzing, and using distractors for multiple-choice tests in education: A comprehensive review. Review of Educational Research, 87(6), 1082–1116.
Gorin, J. S., & Embretson, S. E. (2006). Item difficulty modeling of paragraph comprehension items. Applied Psychological Measurement, 30(5), 394–411.
Graesser, A. C., Singer, M., & Trabasso, T. (1994). Constructing inferences during narrative text comprehension. Psychological Review, 101(3), 371–395.
Haladyna, T. M., & Downing, S. M. (1993). How many options is enough for a multiple-choice test item? Educational and Psychological Measurement, 53(4), 999–1010.
Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied Measurement in Education, 15(3), 309–333.
Halliday, M. A. K., & Hasan, R. (1976). Cohesion in English. Longman.
Hegarty, M., Mayer, R. E., & Monk, C. A. (1995). Comprehension of arithmetic word problems: A comparison of successful and unsuccessful problem solvers. Journal of Educational Psychology, 87(1), 18–32.
Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272.
King, K. V., Gardner, D. A., Zucker, S., & Jorgensen, M. A. (2004). The distractor rationale taxonomy: Enhancing multiple-choice items in reading and mathematics. Pearson Assessment Report.
Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge University Press.
Kirsch, I. S., & Mosenthal, P. B. (1990). Exploring document literacy: Variables underlying the performance of young adults. Reading Research Quarterly, 25(1), 5–30.
Koedinger, K. R., & Nathan, M. J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. Journal of the Learning Sciences, 13(2), 129–164.
Lesh, R., Post, T., & Behr, M. (1987). Representations and translations among representations in mathematics learning and problem solving. In C. Janvier (Ed.), Problems of representation in the teaching and learning of mathematics (pp. 33–40). Lawrence Erlbaum.
Lu, Y., & Sireci, S. G. (2007). Validity issues in test speededness. Educational Measurement: Issues and Practice, 26(4), 29–37.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–103). American Council on Education and Macmillan.
Ni, Y., & Zhou, Y.-D. (2005). Teaching and learning fraction and rational numbers: The origins and implications of whole number bias. Educational Psychologist, 40(1), 27–52.
Parker, M., & Leinhardt, G. (1995). Percent: A privileged proportion. Review of Educational Research, 65(4), 421–481.
Perfetti, C. (2007). Reading ability: Lexical quality to comprehension. Scientific Studies of Reading, 11(4), 357–383.
Recht, D. R., & Leslie, L. (1988). Effect of prior knowledge on good and poor readers' memory of text. Journal of Educational Psychology, 80(1), 16–20.
Rodriguez, M. C. (2005). Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice, 24(2), 3–13.
Schoenfeld, A. H. (1985). Mathematical problem solving. Academic Press.
Smith, J. P., diSessa, A. A., & Roschelle, J. (1993). Misconceptions reconceived: A constructivist analysis of knowledge in transition. Journal of the Learning Sciences, 3(2), 115–163.
Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285.
Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20(4), 345–354.
Thissen, D., Steinberg, L., & Fitzpatrick, A. R. (1989). Multiple-choice models: The distractors are also part of the item. Journal of Educational Measurement, 26(2), 161–176.
Webb, N. L. (1997). Criteria for alignment of expectations and assessments in mathematics and science education (Research Monograph No. 6). National Institute for Science Education.