Synopsis of Potential Errors in the National Crime Victimization Surve (2024)

The mission of the Bureau of Justice Statistics (BJS) is “tocollect, analyze, publish, and disseminate information on crime, criminaloffenders, victims of crime, and the operation of justice systems at all levelsof government” (Bureau of Justice Statistics, n.d.-c). A major tool inaccomplishing this mission is the National Crime Victimization Survey (NCVS),which provides a rich source of information on criminal victimizations, thevictims, and the specifics of their criminal victimizations. In a 2008 report,the National Research Council found that the NCVS provided the only annualsource of national data on criminal victimization with detail at the incidentlevel. The report's first recommendation (3-1) was that “BJS mustensure that the nation has quality annual estimates of levels and changes incriminal victimization” (NationalResearch Council, 2008, p. 3). This panel strongly concurs with thatrecommendation. To further that mission, this panel has focused specifically onthe quality of the measurements of rape and sexual assault with the purpose ofhelping improve quality in BJS's estimates of those types of criminalvictimizations.

The NCVS has both sampling and nonsampling error, as does every survey. The panelexamined the error structure of the NCVS (see Chapters 7 and 8) in considerable detail with the intent of identifying areas thatmight be particular problems. We reiterate that the NCVS is an omnibus surveywith the primary goal of estimating many types of criminal victimization. Itssurvey design is geared toward this broad goal, and it does not and cannotincorporate separate features for measuring each different type ofvictimization. Yet the survey's basic goal and design create problems formeasuring rape and sexual assault. This chapter summarizes the discussions inChapters 7 and 8 that the panel believes are mostproblematic for measuring rape and sexual assault. The chapter concludes withseveral specific recommendations. The bulk of the panel's recommendationsare included in Chapter 10.

POTENTIAL SOURCES OF ERROR: SUMMARY

The NCVS uses a classical area sampling design based on selection in multiplestages to create a sample of housing units. This basic design is used for anumber of general household surveys conducted by the Census Bureau, and itis a good design for an omnibus type survey. For estimates of rape andsexual assault, however, a limitation of the NCVS sample design is itscurrent inability to control the sampling error for estimates of theselow-incidence events. Although the sample size of the NCVS is large, rapeand sexual assault are low-incidence events and the size of the estimatedcoefficients of variation (CVs)^¹ for those estimates at the national level allowsconsiderable year-to-year fluctuation (see Chapter 7). In addition, also because of the lowincidence of rape and sexual assault, BJS has to pool multiple years of datato get estimates of rape and sexual assault for important subpopulations(e.g., by race, gender, marital status, or age group).

In 2011, BJS changed its process for counting series victimizations. Theeffect of the change was substantial for estimates of rape and sexualassault: these victimizations were undercounted in the past, but seriesvictimizations (based on only a few reports) now account for almost 40percent of the national estimates of rape and sexual assault. The handlingof these outlier reports has created additional concern about year-to-yearfluctuation in the estimates of rape and sexual assault.

The overall response rates on the NCVS have remained fairly high, and thereis little item nonresponse coded for specific rape and sexual assaultscreener questions. However, the panel is concerned about panel attrition,particularly because those individuals most likely to dropout—younger people and individuals not living as a couple—maybe at greater risk for rape and sexual assault. The panel's judgmentis that individuals who respond on the NCVS but do not want to report aspecific victimization will refuse the question indirectly by respondingthat no victimization occurred. This item nonresponse would then be coded asa legitimate zero.

Although BJS clearly defines what it intends to measure regarding rape andsexual assault, because of the omnibus design of the screener, theseconcepts are poorly translated through the data collection instruments. Theinstruments use oversimplified terms that can be interpreted in differentways. This situation creates the potential for specification error in theNCVS.

Several issues with the NCVS could lead to measurement error associated withthe questionnaire, the data collection mode, and the interaction betweeninterviewers and respondents. These issues include comprehension of keyquestions and respondents' willingness to answer certain questions.The latter could be affected by the mode of data collection and interviewergender. There continues to be inadequate training of interviewers andmonitoring of the interviews on the NCVS.

All surveys have error associated with their processing. BJS publishes littleinformation about its processing methods or errors found. The lack of suchtransparency makes it difficult for NCVS data users to fully understand NCVSestimates and their limitations.

OBSTACLES TO HIGH-QUALITY ESTIMATES

Based on the potential errors in the NCVS discussed in Chapters 7 and 8 and summarized above, the panelidentified four major obstacles for accurately estimating incidences of rapeand sexual assault.

1.

a sample design that is inefficient for measuring these low-incidenceevents,

Inefficiency of the Sample Design

The NCVS is a general criminal victimization survey that targets thenoninstitutionalized population of the United States with questionsabout different types of criminal victimizations, including crimesagainst property and people. Of the 19 million criminal victimizationsidentified through the NCVS in 2011, 64 percent were property crimes,which are dominated by theft; 29 percent were violent crimes (againstpeople) but not classified as serious; and the remaining 7 percent ofcriminal victimizations were classified as serious violentcrimes,^² ofwhich only 1 of the percentage points was rape and sexual assault (seeFigure 9-1). Thus, from astatistical perspective, rape and sexual assaults are statistically rareincidents.

FIGURE 9-1

Criminal victimizations by type, NCVS 2011. SOURCE: Data from Criminal Victimization 2011(Bureau of JusticeStatistics, 2012a).

The rarity of an attribute in a population presents unique challenges toefficient and effective sampling to estimate the proportion of thepopulation with that attribute. In this situation, the difficulty can beseen by observing the CVs for various types of criminal victimizations,with a lower number meaning a more precise estimate (see footnote 1).Nationally, in 2011, the CV for property crime was 2 percent, forserious violent crime it was 7 percent, and for rape and sexual assaultit was 14 percent (Bureau ofJustice Statistics, 2012a). The NCVS sample design is notwell suited to measure the uncommon attribute of having experienced arape or sexual assault in the past 6 months (see Conclusion 7-1 in Chapter 7).

This concern is intrinsic to the basic design of the NCVS. BJS and theCensus Bureau could address this issue by implementing a very largeincrease to the existing sample size of the NCVS or by modifying theNCVS sample design to focus more efficiently on subpopulations atgreater risk for sexual victimization. (See Chapter 10 for a discussion of such a focuseddesign.) However, the NCVS is an omnibus victimization survey whose goalis to measure a broad range of criminal victimizations. If it were tofocus on the subpopulation at risk for a specific type of victimization(rape and sexual assault), then it would likely make the survey designmuch less efficient for measuring other types of criminal victimizationsacross the whole survey population.

Criminal Context of the Survey

The NCVS is a criminal victimization survey. It is introduced that way tohousehold members, and the questionnaire goes through a listing ofdifferent types of crimes, asking each respondent whether any of thesecrimes has happened to him or her. As discussed in Chapter 8, victims may notalways think of a rape or sexual assault as a crime, particularly if therespondent knows the offender. In addition, some victims may feardisclosure to police and may associate a government crime survey tooclosely with law enforcement. The panel concludes that the context of acriminal victimization survey is a major barrier to accurate reportingof incidents of sexual victimizations (see Conclusion 8-7 in Chapter 8).

BJS could address this issue by moving the measurement of rape and sexualassault to a separate survey (as recommended in Chapter 10) or by changingthe entire context of the NCVS to something more neutral. Themodification of the NCVS into a survey that has a “neutralcontext” would entail a major redesign of that survey (includingits name), and such a change would undermine other goals of the survey.The context of criminal victimization appears to work well for reportingother, less sensitive, victimizations. A redesign of the NCVS to removethe “criminal context” on this large omnibus survey wouldmake it difficult to effectively communicate the purpose of the surveyto respondents, and it would likely negatively affect the reporting ofmany other types of victimizations.

Lack of Privacy in Responding

Providing a respondent with privacy is an important prerequisite for anysurvey that deals with sensitive questions. Privacy, specifically fromother household members, is critical for accurately responding toinquiries about rape and sexual assault, in part because the victimoften knows the offender. In fact, the offender may be a householdmember. The current NCVS data collection protocols do not providesufficient privacy (see Conclusion 8-8 in Chapter 8). Some privacy gains would be possible onthe NCVS by switching to a self-administered mode of data collection.This switch would be a major change for the survey, and it would likelyhave both beneficial and detrimental effects on collecting informationabout other types of criminal victimizations.

The panel concludes that such a change of data collection mode isimportant to make, but that this change alone would still not providethe level of privacy that is needed for reporting sexual victimizations.As long as multiple members of the household are interviewed with anidentical questionnaire, an adequate level of privacy is unlikely to beobtained. Each household member knows the questions that are asked onthe survey and the duration of the interview based upon their ownresponses. An individual household respondent may continue to haveconcerns about reporting an incident of rape or sexual assault underthose circ*mstances. To achieve the privacy level needed for questionsregarding rape and sexual assault, the survey protocol would also needto change to targeting only a single person in each household or todeploy a “matrix” design in which each household memberwould be screened for different types of criminal victimizations. Ifimplemented for the NCVS as a whole, then either approach would severelyreduce the number of responses obtained for each specific type ofvictimization. Without an accompanying increase in sample size, thischange would negatively affect the precision for estimates of all typesof victimizations.

Ambiguous Terms in the Questionnaire

The current NCVS screening questionnaire uses a series of cues to aidrespondents in remembering past victimizations. As part of those cues,it uses the terms rape and sexualassault without any further elaboration as to what thoseterms mean. Although BJS has precise meanings for these terms, they arenot provided to respondents. It is not reasonable to assume that allindividual respondents will interpret these words identically or as BJSanticipates (see Conclusion 8-6 in Chapter 8). More behaviorally specific words that describe aspecific set of actions would lead to more accurate responses on thesequestions.

Such a behavioral approach is not completely alien to the NCVS. Some ofthe screening questions for assault do use somebehaviorally specific language: a respondent is not asked whether he orshe has been assaulted, but is asked questions that describe specificbehaviors—“Has anyone attacked or threatenedyou … with anything like a baseball bat, frying pan,scissors, or stick?” It would be possible to makebehaviorally specific changes on the current NCVS regarding rape andsexual assault with only a modest effect on other parts of thequestionnaire. However, there would need to be specific screening forrape and sexual assault that is separate from and in addition to themore generalized cuing sequences currently used in the screener. Anotherdifficulty is that the NCVS interviews children as young as 12 years ofa*ge, and there may be parental objections to the behaviorally specificquestions. With a separate survey as recommended by the panel, therewould be more options available to BJS on handling respondents betweenthe ages of 12 and 17.

GENERAL RECOMMENDATIONS

The majority of the panel's recommendations specifically address newdirections for measuring rape and sexual assault (see Chapter 10). In this section, thepanel offers several general recommendations to BJS regarding improveddocumentation and future research that grew out of the panel's reviewof the NCVS. With regard to research, we consider the unaddressed topic ofchild victimization.

The error review contained in Chapters7 and 8 andsummarized above identifies potential errors on the NCVS. The panel did nothave the time or resources to conduct a complete error profile of the NCVS,which would include measuring the levels of error and their effects on theestimation of rape and sexual assault. The panel recommends this action toBJS.^³

RECOMMENDATION 9-1 The Bureau of Justice Statistics should conductan in-depth total error profile of the National Crime VictimizationSurvey, specifically focusing on estimation of rape and sexualassault. This profile should be made available to public datausers.

Throughout its work, the panel found that the publicly availabledocumentation was sometimes difficult to locate on the BJS website, and themajor methodology document was several years out of date. The age and lackof clarity of existing documentation of the NCVS inhibits a complete andaccurate understanding of the NCVS methodology.

RECOMMENDATION 9-2 The Bureau of Justice Statistics should providecomplete documentation of the methodology in its surveys, details ofsurvey response, costs, and the individual components of totalsurvey error. The documentation should be made publicly availableand easy to access.

Although BJS has commissioned a number of research studies over the past 10years and many of the final reports of those studies are on the BJS website,they are inadequately organized to allow data users easy access. Aninterested data user has to know that a particular research study wasconducted and then do a “hit or miss” search to see whether aresearch report on that topic was posted. In addition, the documentSurvey Methodology for Criminal Victimization in the UnitedStates (Bureau ofJustice Statistics, 2008b) has been periodically updated, but themost recent update appears to be 2008.

RECOMMENDATION 9-3 The Bureau of Justice Statistics should clearlylay out its research agenda on the agency's website and linkthe agenda topics to completed reports. For research in progress,the website should include information about what is being done andwhen a report is expected.

The NCVS and this report do not address the victimization of children underthe age of 12 years, a serious and neglected problem in the overallmeasurement of sexual victimization. There is a partial accounting of thevictimization of children because several of the publichealth—oriented surveys that measure lifetime prevalence of rape andsexual assault have asked respondents their age when first victimized. Forexample, the National Intimate Partner and Sexual Violence Survey estimatedthat of individuals who reported having been raped in their lifetime, 12.3percent of females and 27.8 percent of males were first raped before the ageof 12 (Black et al.,2011).

The panel believes that the victimization of young children is a seriousproblem that requires further attention. No ongoing survey captures thisinformation, with child protective services providing the major source ofinformation on these victimizations. Unfortunately, this situation issimilar to when the Uniform Crime Reports was the single source reporting onthe sexual victimization of adults, with resulting concerns aboutunderreporting to authorities. However, the issues associated withinterviewing children mean that the measurement of this type ofvictimization needs a specialized approach—it cannot be measuredthrough the same vehicle and the same methods that are used for adults andolder children. The panel did not have the time and resources to pursue thisextensive area of research. Instead, it recommends that BJS explore optionsto measure these sexual victimizations in the future. This might include theinclusion of retrospective questions to adults about childhoodvictimizations.

RECOMMENDATION 9-4: The Bureau of Justice Statistics should beginresearch to explore options for measuring the incidence of rape andsexual assault of children younger than 12.

1: The CV is the standard error of a survey estimate divided by the estimateitself (expressed as a percent). It provides a relative measure of thesampling error associated with survey estimates.
2: Serious violent crimes include robbery and aggravated assault, but notmurder and kidnapping, which are not estimated through the NCVS.
3: To provide guidance in implementing this recommendation, here are someexamples of error profiles in the literature: Brooks and Bailar (1978);Chakrabarty and Torres (1996); Doyle and Clark (2001); Jabine, King, andPetroni (1990); Kalton et al. (2000); and U.S. Energy InformationAdministration, 1996. Kasprzyk and Kalton (2001) review the use ofquality profiles in U.S. statistical agencies and discuss theirstrengths and weaknesses for survey improvement purposes.