More Procrastination

The recently published reports on the 2003 NAAL can be quite interesting. In particular, the "Web Documentation for NAAL General Audience Report" provides information on the sampling and statistical processing. It also demonstrates just how difficult to comprehend a combination of prose and quantitative information can become, and leads this writer to wonder about his own level; try this:

Based upon the screener data, 23,732 respondents aged 16 and older were selected to complete the background questionnaire and the assessment. Of the 23,732 household respondents selected, 18,186 completed the background questionnaire. Of the 5,546 respondents who did not complete the background questionnaire, 355 were unable to do so because of a literacy-related barrier; either the inability to communicate in English or Spanish (the two languages in which the background questionnaire was administered) or because of a mental disability. The final weighted response rate for the background questionnaire, including respondents who completed the background questionnaire and respondents who were unable to complete the background questionnaire because of language problems or a mental disability, was 75.6 percent.

I wonder what the weighting for the final response rate was. It appears to relate 18,541 (18,186 + 355) to something, but to what? At any rate, 21.9% of the respondents selected to complete the background questionnaire and the assessment did not complete it, for reasons other than inability to communicate with the interviewer.

This is neither the first nor the last difficulty for the administrators. The first was to find households. Beginning with a nationally representative probability sample of 35,365 households (chosen from what database?), they encountered fully 13.2% (4,671 ) "were either vacant or not a dwelling unit". Subsequent screening (and representativity targets) then pared the group down to the 23,732 noted in the quoted paragraph above. Another five per cent of those tested did not properly complete all three scales. Amusingly, the prison inmates tested seem to have been more successful at staying in the sample and completing all three scales.

If I read their report correctly, the (non-prison) sample yield breakdown is this:

	18,186	adults age 16 or older completed the background questionnaire,
of whom	17,178	completed at least one literacy task on each of the three scales—prose, document, and quantitative—included on the assessment.
That leaves	1,008	who completed the background questionnaire but not the full core test.
Deducting	84	who were disqualified for language or mental reasons during the test (i.e., after the background questionnaire)
leaves	924	who completed some, but not all three scales.
		One accounting given is
	859	respondents answered the background questionnaire but refused to complete the assessment for reasons other than language issues or a mental disability. For these respondents, answers to one assessment item on each scale were imputed based upon the answers from respondents with similar background characteristics.
plus	65	a wrong response on each scale was imputed for respondents who started to answer the assessment but were unable to answer at least one question on each scale because of language issues or a mental disability.
		The aforementioned 859 would then have included
an additional	504	were unable to answer at least one question on each of the three scales for literacy-related reasons.
so we surmise that	355	just plain had no time to waste on such foolishness.

Next in this series: why this test is better than the OECD PISA test, and how it correlates to SAT verbal scores.

---

18,102 + 439 = 18,541 : final reporting sample plus language and mental disqualifications.

18,186 + 355 = 18,541 : completed background questionnaire plus language and mental disqualifications.

Ergo, 84 language and mental disqualifications were manifest during the assessment but not during the background questionnaire.