## SAT Won’t Report Low Scores

National Review’s blog reports that the SAT is changing so that only a student’s MAXIMUM score out of all the times he takes the test will be reported to colleges. What amazing favoritism to rich, stupid, applicants!

Or maybe not so amazing. This will be a bonanza for the SAT company, since their tests will be taken so many more times. This is especially true nowadays, when many colleges have merit-based scholarships and your $45 retest fee might have a 1/10 chance of yielding you $1000 extra in tuition breaks.

It also raises an interesting mathematical question. Suppose everyone ends up taking the test exactly 8 times. This will cost a lot more, of course, but will it yield more accurate evaluation of the applicants? Which provides more useful information:

1. A single test score.

2. The maximum of 8 test scores.

The answer depends on the distribution of an individual’s test scores for his given talent. If someone with ability X scores X on the test with probability .9 and X-y with probability .1, the Maximum is a better measure (in fact, then it is even better than the average of 8 test scores).

If someone with ability X scores X on the test with probability .8, X-y with probability .1, and X+y with probability .1, which is better? The maximum still, I think. In almost every case, person i will end up with a maximum of Xi+y, and we can simply subtract y and get a person’s ability.

If someone with ability X scores X on the test with probability .999 and X+y with probability .001, then I think , the Single reported score is better. It is right with probability .999, whereas the Maximum will frequently be X+y (with probability 1-.999^8) so it will be right with only probability .992. (I haven’t phrased that carefully– what we care about is not the percentage of “right” answers but the variance of the measure minus the true ability, but in this special case the two criteria give the same answer.)

What if the distribution of test scores around ability has a normal distribution? I don’t know. The answer might depend on the variance. I’ll ask our job candidate at lunch. He’s a couple of years out of grad school already, so he shouldn’t freak out at the question.

So, I’d like to take this apart a bit. However, I would like to start with a statement: Yes, this favors wealthy students, probably unfairly, and enriches the College Board, also unfairly.

There are several phenomena at work, but the biggest problem is that scores tend to go up as the test is repeated. This is the virtue of all those practice tests in test prep classes. This means that it isn’t a simple normal distribution; there is even a bias over time. Of course, there are also underlying variances in performance based on the random nature of the question selection and other performance variables (feeling that day). Finally, there is a very serious issue with students having target scores and taking the tests until those targets are achieved. This will mean that there are all sorts of interesting violations of normality as people, on average, creep up to the score they want, as long as they have time and keep paying.