The dilemma of survey nonresponse

Library&InformationScienceResearch

25(2003)239–263

Thedilemmaofsurveynonresponse

JacquelynBurkell

FacultyofInformationandMediaStudies,MiddlesexCollege,

TheUniversityofWesternOntario,London,OntarioN6A5B7,Canada.

E-mailaddress:jburkell@uwo.ca(J.Burkell).

Abstract

Anexaminationofthelibraryandinformationscience(LIS)literaturerevealsthatsurveyspub-lishedfrom1996through2001inthreemajorLISjournalshaveanaverageresponserateof63%,andalmostthreefourthsofthesurveyshavearesponseratelessthan75%(thelevelthatiswidelyheldtoberequiredforgeneralizability).Consistentwiththepracticeinotherdisciplines,however,mostLISresearchersdonotaddresstheissueofnonresponsebeyondreportingthesurveyresponserate.ThisarticledescribesastrategythatLISresearcherscanusetodealwiththeproblemofnonresponse.Asafirststep,theyshouldusemethodologicalstrategiestominimizenonresponse.Toaddressnonresponsethatremainsdespitetheuseofthesestrategies,researchersshoulduseoneofthefollowingstrategies:carefuljustificationofadecisionsimplytointerpretsurveyresultsdespitenonresponse,limitingsurveyconclusionsinrecognitionofpotentialbiasfromnonresponse,orassessingandcorrectingforbiasfromnonresponse.

D2003ElsevierInc.Allrightsreserved.

1.Introduction

Libraryandinformationscience(LIS)researchersstudytheneeds,challenges,andproblemsofinformationprofessionalsandinformationusers,andsurveysareanexcellentmethodofcollectinginformationabouttheopinionsandexperienceofresearchparticipants.SurveysarethereforeamongthemostwidelyusedmethodsinLISresearch(Blake,1994;Callison,1997;Dimitroff,1995;Feehan,Gragg,&Havener,1987;Fidel,1993;Goodall,1996;Julien,1996;Julien&Duggan,2000;McKechnie,Baker,Greenwood,&Julien,2002;Preitz,1980–81;Simpson,1992).GiventheprevalenceofsurveysasamethodofcollectingdatainLIS,issuesofsurveymethodologyareofparamountimportancetoLISresearchers.

0740-8188/03/$–seefrontmatterD2003ElsevierInc.Allrightsreserved.doi:10.1016/S0740-8188(03)00029-X

240J.Burkell/Library&InformationScienceResearch25(2003)239–263

SpecificissuesthathavebeenaddressedintheLISliteratureincludeoptimizationofsampledesign(Lakner,1998)andtheutilityofe-mailremindersforsurveyssentbyregularmail(Roselle&Neufeld,1998).HernonandSchwartz(2000)raisedanothercriticalissueinsurveymethodology:theproblemofnonresponse.Theyremarkedthat,intheLISliterature,‘‘insufficientattentionhasfocusedonreturnratesandwhetherthepopulationistrulyrep-resented’’(p.119),andasked,‘‘Whatcanbedonetogetasufficientnumberofrespondents?’’and‘‘Howcanweframetheimpositiontogaincooperation?’’(p.118).Thisarticleaddressesthesequestionsandotheraspectsoftheproblemofnonresponse.

Well-designedsurveysareeffectiveresearchinstrumentsforgatheringquantitativedata.Whenthosedataarecollectedfromacarefullyselectedrepresentativesampledrawnfromalargerpopulation,theresultscanbegeneralizedbeyondthesurveyrespondentstotheentirepopulationofinterest.Thereis,however,arecurrentproblemencounteredbyvirtuallyallsurveyresearchersthataffectsthisgeneralization:peoplewhoareaskedmaynotrespond.Studentsofpsychologyareintroducedtotheconceptofthe‘‘blindspot,’’anareaoftheretinathatreceivesnodirectvisualinput.Ourvisualsystemrespondsby‘‘fillingin’’themissingdata,relyingoninformationfromsurroundingareastoprovidea‘‘bestguess’’aboutwhatappearswherewecannotactuallysee.Thisinductiveprocessworkswellinasmooth,continuousworldwheretheseenisagoodapproximationoftheunseen.Theprocessfails,however,whennewandunexpectedvisualinformationiscontainedcompletelywithintheboundariesoftheblindspot.Inthiscase,startlingandpotentiallyimportantinformationgoescompletelyunnoticed.

Surveyresearchershaveananalogousblindspottocontendwith:themissingdatathatresultfromnonresponse.Theresearcherreceiveslittleornoinformationfromnonrespon-dentsandisfacedwiththechallengeofformingacompletepictureofthesurveyedpopulationfromincompletedata.Thesolutionmostoftenadoptedissimplytofillinthegapwithinformationcollectedfromrespondents,assumingthattheirdataapproximatesthedatamissingfromnonrespondents.Thissolutionworkswellifnonrespondentsdonotdifferfromthosewhoprovidedata,butitfailsifthetwogroupsaredifferentinwaysthatinfluencetheirsurveyresponses.Whensurveyresultsaregeneralizedtothepopulationinthelattercase,itisasifthenonrespondentsdonotexist:thepopulationthatisdescribedbythesurveyresultsisthepopulationofrespondentsonly.

Theconsequencesofnonresponseforconclusionsbasedonsurveydatacanbeserious.Librarypatronsatisfactionwouldbeoverestimatedifthoselesshappywithlibraryserviceswerealsolesslikelytorespondtoasatisfactionquestionnaire.Librarianworkloadwouldbeunderestimatedifthosebusiestintheirworkcouldnotfindthetimetocompletethesurvey.Academiclibrariesfacedwiththedifficultdecisionofwhethertodiscontinuejournalsubscriptionsmightuseasurveytocollectpatroninputonthequestion,onlytobemisledabouttheneedsofthegeneralpopulationifthosewhodonotusethejournalssimplydonotrespondtothesurvey.Ineachofthesecases,thedatafromnonrespondentswouldhavechangedthesurveyconclusions.

Nonresponseintroducesanunresolvabledilemma:ultimately,assessmentof,andcorrec-tionfor,nonresponserequiresthatresearcherseitherestimatedatafororextractdatafromthoseindividualswhohavenotprovidedanydata.Thisdilemmaisacentralissueinsurvey

J.Burkell/Library&InformationScienceResearch25(2003)239–263241

methodology.In1999,anentireconference(theInternationalConferenceonSurveyNonresponseinPortland,Oregon)wasdevotedtononresponse(Groves,Dillman,Eltinge,&Little,2002).Itiswidelyheldthataresponserateof75%to90%issufficienttosup-portgeneralizationsfromthesurveyedsampletothepopulationofinterest(Ary,Jacobs,&Razavieh,1996;Gall,Borg,&Gall,1996;Kerlinger,1986;Tuckman,1999).Overall,however,onlyabout70%ofthoseapproachedagreetocompleteasurvey,andthereisevidencethatnonresponseratesareincreasing(Brehm,1993).Surveysofsomegroupsshowevenlowerresponserates:forexample,approximately60%ofphysiciansrespondtosurveyrequests(Cummings,Savitz,&Konrad,2001;Sibbald,Addington-Hall,Brenneman,&Freeling,1994).LISresearchersalsofacetheproblemofnonresponse.AnexaminationofLISsurveyresearch,reportedinmoredetaillater,revealsanaverageresponserateof63%.Thus,mostsurveyspublishedinLISandinotherdisciplinesshowsomedegreeofnon-response,andmanyhaveresponseratesthatdonotmeetthelevelrequiredforgeneralizingbeyondthoseindividualsactuallysurveyedtothepopulationofinterest.

Surveynonresponseisaubiquitousproblem.Buthowshouldresearchersdealwiththeissue?Mostresearchmethodstextbooksprovidelittle,ifany,guidance.Amongthefewtextbooksthatnotetheproblemofnonresponse,somediscussmethodologicalvariationsthatimproveresponserates(e.g.,Neuman,2000)andasmallernumberprovidegeneralguide-linesabouthowtoassesstheimpactofnonresponse(e.g.,Palys,1997).Detaileddiscussionsofstatisticalapproachestoassess,andpossiblycorrectfor,nonresponseareusuallylimitedtospecializedworksinsurveymethodology(e.g.,Grovesetal.,2001).ThesituationisnobetterwithinthedisciplineofLIS:ofthreemethodstextbookswidelyusedinLIS(Busha&Harter,1980;Losee&Worley,1993;Powell,1997),onlyone(Losee&Worley,1993)mentionstheissueofnonresponse,andthetreatmentinthattextiscursory.LISsurveyresearchers,therefore,arelargelyleftontheirownwithregardtononresponse.Theyknowthereisaproblem,buttheyarenotexactlysurewhattodoaboutit.

ThisarticleaddressesthedilemmaofnonresponsebyprovidingageneraloverviewoftheproblemandadetaileddiscussionofthevariousapproachesthatLISresearcherscanusetominimizethedegreeofnonresponseandtheimpactofnonresponseonsurveyresults.Thefinalsectionofthearticleprovidesanoverviewofthetreatmentofnon-responseintheLISliterature.ThisoverviewisbasedonanexaminationofsurveyspublishedinthreeprominentLISjournalsfrom1996through2001andidentifyingthedegreeofnonresponseinthesesurveysandthedegreetowhichresearchersusestrategiestoaddressthenonresponseproblem.

2.Whatisnonresponseandwhyisitimportant?

Surveynonresponsereferstothediscrepancybetweenthegroupapproachedtocompleteasurveyandthosewhoeventuallyprovidedata.Despitethebestintentionsandeffortsofresearchers,dataarerarelycollectedfromeachmemberoftheidentifiedsample,andmostsurveysthatachieveaperfectresponserate(especiallythosewithlargerandmorediversesamples)dosoatsignificantcostintermsofresearchereffortandfinancialresources.Inmost

242J.Burkell/Library&InformationScienceResearch25(2003)239–263

surveys,alargeproportionofthoseindividualschosenforthesampledonotparticipateeitherbecausetheattempttocontactthatindividualisunsuccessfulorbecausetheindividualfailstocompletethesurvey.

Althoughthecausesandconsequencesofnonresponsedifferfromsurveytosurvey,somegeneralconclusionsaresupportedbypreviousresearch.Thesestudiesprovidesomeinsightintotheprofileofsurveyrespondents(comparedwithnonrespondents),allowingresearcherstomakeeducatedguessesregardingdifferencesbetweenrespondentsandnonrespondentsandtohelpthemmakechangesinsurveyadministrationtoimproveresponserates.

Thoseindividualswhorespondtoasurveytendtobemoreinterestedinthetopicofthesurveyand/ormoreinterestedintheactivitiesbeingstudied(Donald,1960;Martin,1994;Senf,1987).Inaddition,respondentsaremorelikelytobelievethatsurveyresponses(includingtheirown)willbeusedtomakechanges(Rogelburg,Luong,Sederbury,&Cristol,2000).Individualsgivethefollowingreasonsfornotreturningmailsurveys:theydidnotreceiveit,theyweretoobusy,theyforgotorlostit,theywerenotinterestedinthetopic,thesurveywastoolong,ortheythoughtitwasnotintendedforthem(Robinson&Agisim,1951;Sosdian&Sharp,1980).Theprofileofrespondentsinsurveysofthegeneralpublicmatchesthatofpubliclibraryusers(Berelson,1949):respondentstendtobefemale,older,andmoreeducatedthannonrespondents(Green,1996;Kaldenberg,Koenig,&Becker,1994).

Organizationalrepresentativescitereasonsforrefusaltocompleteasurveythataresimilartothosecitedforindividualnonresponse.Thesereasonsincludeinconvenience,inappropriateorirrelevantsubjectmatter,concernabouttheconfidentialityoftheinformationprovided,timeconstraints,surveylength,andlackofbenefittotheorganization(Albaum,Evangelista&Medina1998;Baldauf,Reisinger,&Moncrief,1999;Diamantopoulos&Schlegelmilch,1996;Tomaskovic-Devey,Leiter,&Thompson,1994).Whenorganizationsarethepopula-tionofinterest,organizationalsizeisanimportantdeterminantofresponse:largerorga-nizationsarelesslikelytorespondtosurveys(Cotton&Wonder,1982;Tomaskovic-Deveyetal.,1994).

Nonresponsepresentstwoproblemsfortheinterpretationofresearchresults.First,itreducesthesamplesizeand,thus,decreasestheprecisionwithwhichresultscanbestated.Second,andmoreimportant,isthefactthatitintroduceserrorintothesamplingprocessbyexcludinganonrandomsubsetofthepopulation.Iftheexcludedsubsetisdifferentfromthoseremainingwithregardtothesurveydata,theresultsofthesurveywillbebiased.Thefirstissue,ifitoccursalone,isneitherparticularlyseriousnorparticularlydifficulttoremedy.Inmostcases(exceptwhereacensushasbeenattempted),additionalrespondentscanberecruitedinnumberssufficienttocompensateforexpectedrefusals.Ifthisstrategyisnotfeasible,statisticaltechniquesautomaticallyaccountforsmallersamplesizebyincreasingtheconfidenceintervalsaroundestimatesofpopulationparametersorbydecreasingthesignificanceofinferentialstatistics.However,bothoftheseresponsestoasmallersamplearevalidonlyiftheresultsareunbiased,thatis,respondentsarerepresentativeoftheentirepopulation.Whennonrespondentsdiffersignificantlyfromsurveyrespondents,increasingthesizeofthesampleorusingstatisticsthatreflectsamplesizewithoutaddressingthebiascausedbynonresponseservesonlytobolsterreaderconfidenceinpotentiallyinaccu-rateresults.

J.Burkell/Library&InformationScienceResearch25(2003)239–263243

Nonresponsealwaysintroducessomebiasintothesample,evenifthatbiascannotbedetectedbystatisticaltestsonavailablecomparisonmeasures.Nonresponsearisesdirectlyasaresultofanindividual’schoice(e.g.,thechoicenottocompletethesurvey)oranindividual’scharacteristics(e.g.,thecharacteristicofhavingmovedandthereforehavinganinvalidmailingaddress).Inthesechoicesand/orcharacteristics,nonrespondentsbydefinitiondifferfromrespondents.Thegroupsalsodifferinthequalitiesthatdeterminethechoicesand/orarecorrelatedwiththecharacteristics.Thus,publiclibrarypatronswhochoosenottocompleteasurveyregardingtheimportanceofInternetaccessarealsolikelytousetheserviceless,andthegroupoflibrarianswhodonotreceiveajobsatisfactionsurveybecausetheyhavechangedpositionsislikelytoincludemanywhochangedjobsbecausetheyweredissatisfiedwiththeirwork.Occasionally,researchersarguethatiftheycannotidentifyanysystematicdifferencebetweenrespondentsandnonrespondents,thereisnobiascausedbynonresponse.Statisticalcomparisonsofrespondentsandnonrespondentsonavailabledescriptors,however,canonlyeliminatethepossibilitythatthetwogroupsdiffersignificantlyintermsofthemeasuredqualities.Itremainspossible,andevenlikely,thattherearedifferencesbetweenthegroups:eitherrealdifferencesonmeasuredvariablestoosmalltoreachstatisticalsignificanceordifferencesonothervariablesforwhichdataareunavailable.

Thequestion,therefore,isnotwhethernonresponsehasresultedinabiasedsample:thatanswerisalways‘‘yes.’’Theimportantissueiswhetherthebiasinfluencessurveyresults.Thereis,however,adearthofdirectresearchonthisquestion,foragoodreason:thedegreetowhichnonresponseaffectssurveyconclusionsdependsontherelationshipbetweenthevariablesofinterestandthecausesofnonresponse(Brehm,1993),andthisrelationshipislikelytodifferfromsurveytosurvey.Ifthereisnorelationship,nonresponsewillnotaffectthesurveyresults.If,however,thereisarelationship(eitherdirectorthroughathirdvariablethatinfluencesbothnonresponseandthedependentvariablesofinterest),nonresponsewillcompromisethesurveyconclusions.Ifresearcherswereabletomeasuretherelationshipbetweenselectionmechanismsandsurveyresultsfortheirparticularsurvey,itwouldbesimpletodeterminewhethernonresponserepresentedathreat.Thenatureofthatrelationship,however,isvirtuallyimpossibletodetermine,becauseassessingitrequirestheverythingthatnonresponseprecludes:analysisofdatafromnonrespondents.

Thisisthecentraldilemmaofnonresponse:theimpactofnonresponseonsurveydatacannotbedeterminedwithoutdata(eitheractualorestimated)fromnonrespondents.Nonresponseraisesthespectre(butnotthecertainty)ofbiasedresults;furthermore,thehigherthelevelofnonresponse,thegreaterthepotentialbias(Alexander,Alliger,&Hanges,1984;Chen,1996;Cochran,1963;vanGoor&Stuiver,1998;Viswesvaran,Barrick,&Ones,1993).Inisolatedcases,researchershavedemonstratedthatlowresponseratesdonotnecessarilycompromisesurveyresults(Keeter,Miller,Kohut,Groves,&Presser,2000;Visser,Krosnick,Marquette,&Curtin,1996).Thisfindingshouldnotbetakenasgeneralevidencethatnonresponsecanbeignored,however.Theimportantissueisrepresentative-ness:whethertherespondentsresemble(withregardtosurveyresults)thepopulationfromwhichtheyweredrawn(Cook,Heath,&Thompson,2000;Cook&Thompson,2001;Krosnick,1999;Thompson,2000).Itispossibletointerpretsurveyresultsevenwithhigh

244J.Burkell/Library&InformationScienceResearch25(2003)239–263

levelsofnonresponseifitcanbearguedthattherespondentsarerepresentativeofthepopulation.Forexample,CookandThompson(2001)arguedthat,despitearesponserateofapproximately14%(Thompson,2000),theresultsofaWeb-basedLibQUAL+studycouldbegeneralizedtothelargerpopulationbecausetherespondentswereshownnottodiffersignificantlyfromthepopulationonvariousmeasuredvariables(Thompson,2000).TheapproachusedbyCookandThompsonisonestrategyfordealingwithnonresponse.Thenextsectionofthisarticlepresentsthisandothersuggestionsforaddressingthedilemmaofnonresponse.

3.Dealingwiththedilemmaofnonresponse

Asafirststrategyfordealingwithnonresponse,theresearchershouldaddresstheproblematthesourcebytakingadvantageofmethodologicalvariationsproventoreducenonresponse.Althoughtheseeffortsareimportant,inmostcasestheywillnotresultina100%responserate.Theresearcheristhereforeleftwiththeproblemofinterpretingsurveyresultsinthecontextofsomeremaininglevelofnonresponse.

Reportsofsurveyresultsshouldalwaysindicatethedegreetowhichinitiallyselectedrespondentsparticipateinthesurveybyreportingtheresponserate(responserate=1ànonresponserate).Thereareseveraldefinitionsofresponserate,whichdifferinthewaythatincompletesurveys,noncontacts,refusals,andotheroutcomesaretreated(AmericanAssociationforPublicOpinionResearch,2000).Thisarticleusesthe‘‘maximumresponserate’’definedbytheAmericanAssociationforPublicOpinionResearch:responserate=(completeresponses+partialresponses)/totalnumberintheeligiblesample.Whentheresponserateislessthan100%,theresearcherhasthreechoices:(1)ignorethenonresponse,preferablyonasubstantivebasis(e.g.,alowlevelofnonresponseordocumentedreasontobelievethatnonresponsewouldnotaffectresearchresults);(2)limitthesurveyconclusionsbasedontherateandpresumedimpactofnonresponse;or(3)attempttoassessand,ifnecessary,correctforthesamplingbiasintroducedbynonresponse.3.1.Addressingtheproblematthesource:Methodstoreducenonresponse

Themethodofsurveyadministrationisanimportantdeterminantofresponserates.Fourmethodsthatarewidelyusedinsurveysaremail,telephone,face-to-face,ande-mailadministration(somesurveysarealsodistributedbyfax,butthesearefew,andthemethodhasquicklybeensupplantedbye-mailadministration).SurveyshavealsobeendeliveredovertheWeb(e.g.,Perkins&Yuan,2001),butWebsurveysofprobabilitysamplesgenerallyrequirerecruitmentofrespondentsusingoneoftheothermeansofadministration(Couper,2000).Ofthefourmethodswidelyusedwhenarepresentativesampleisdesired,face-to-faceadministrationtendstoresultinthehighestresponserates(Hox&Deleeuw,1994),althoughthiseffectmaydifferacrossdemographicgroups(Krysan,Schuman,Scott,&Beatty,1994).Telephonesurveysdemonstratethenexthighestresponserates,followedbyregularmailandthene-mail(Hox&Deleeuw,1994;Kettleson,1995).

J.Burkell/Library&InformationScienceResearch25(2003)239–263245

Althoughresponseratesarelowestformailande-maildelivery,thesemethodsarealsotheleastexpensiveandthereforearewidelyusedinsurveyresearch.Dillman(2000)hasdevelopeda‘‘tailoreddesignmethod’’toreducetheerrorassociatedwithnonresponse(amongotherfactors)inmailande-mailsurveys.Hesuggestedthatsurveyresponsecanbemaximizedbythefollowing:(1)establishingtherespondent’strust,(2)increasingtheexpectedrewardsofparticipation,and(3)reducingthesocialcostsofparticipation.SomeofthespecificstrategiessuggestedbyDillmanarerelativelylow-costandsimpletoimplementinanysurvey.Thesestrategiesincludeprenotification,personalizedcoverletters,theuseofreminders,incentiveswiththeinvitationtoparticipate,andstamped,self-addressedenvelopesforthereturnofmailsurveys.Researchsuggeststhat,ofthesemethods,smallmonetaryincentives(i.e.,$1to$2)includedwiththeinitialsurveymailoutarethesinglemosteffectivestrategythatresearcherscanusetoincreaseresponserates(Church,1993;Fox,Crask,&Kim,1988;Gajraj,Faria,&Dickinson,1990;Helgesen,Voss,&Terpening,2002;Hopkins&Gullickson,1992;Singer,VanHoewyk,&Maher,2000).Meta-analysesofstudiesofmailsurveyresponserateshaveofferedsupportforthestrategiessuggestedbyDillman(Foxetal.,1988;Hopkins&Gullickson,1992;Yammarino,Skinner,&Childers,1991),indicatingthattheimpactofinterventionstoincreaseresponseratesisconsistentacrosspopulations(Green,Boser,&Hutchinson,1998).Hart(1998)usedmanyofDillman’ssuggestionsforasurveyoftherelationshipsbetweenworkrolesandinformationgatheringforcollegefacultyandachievedaresponserateof84%.

Inadditiontothesestrategies,researchersshouldconsiderselectingasmallerinitialsampleandconcentratingeffortsandresourcesonachievingahighresponserate(Wayne,1975–76).Researchersmaychoosetouseoneofthemoreexpensivemethodstocollectdatafromthesmallersample(e.g.,telephoneinterviews),theymaychoosetousemultiplemethodsinsolicitingresponse(Roselle&Neufeld,1998),ortheymayprovideincentivestothoseinvitedtoparticipate.Althoughasmallersamplelimitstheprecisionwithwhichresultscanbestated,thisdisadvantageisoffsetbythereductioninbiasassociatedwithanincreasedresponserate.3.2.Ignoringnonresponse

Whencanyousimplyignorenonresponse?Theshortanswer,correctinthemostabsolutesense,isnever.Althoughlowresponseratesdonotnecessarilycompromisesurveyresults(Keeteretal.,2000;Krosnick,1999,Visseretal.,1996),anylevelofnonresponsecouldbiasconclusions(Jones,1996).Therealproblemisthattheresearchercannotknowwhethernonresponseaffectssurveyconclusions,becausethisrequiresinformationaboutthediffer-encesbetweenrespondentsandnonrespondents.Nonetheless,someconditionscanjustifyignoringthenonresponse,andthefactorsthatshouldbetakenintoaccountareoutlinedinthissection.Researchersshouldnot,however,seethisasapanaceaordefaultstrategyfordealingwithnonresponse.Ignoringtheproblemisappropriateonlyinasmallproportionofcases,andtheresearchermustensurethattheuseofthestrategyisjustified.

First,theimpactofnonresponseonsurveyresultsdependsentirelyontherelationshipbetweenthemechanismsthatresultinnonresponseandthevariablesofinterest.Again,bydefinition,theresearcherisnotinapositiontomeasurethisrelationshipbecausetherequired

246J.Burkell/Library&InformationScienceResearch25(2003)239–263

dataareunavailable.Itispossible,however,tohypothesizeabouttherelationship.Totheextentthattheresearchercanmakeastrong,logicalcasethatthelikelihoodofresponseisunrelatedtosurveyconclusions,thereissomejustificationforignoringnonresponse.

Second,thedegreeofnonresponseisimportant.Potentialbiasincreasesmonotonicallywithnonresponse(Alexanderetal.,1984;Chen,1996;Cochran,1963;vanGoor&Stuiver,1998;Viswesvaranetal.,1993):thehigherthelevelofnonresponse,thegreaterthepotentialbias.Giventhatanydegreeofnonresponsecanleadtobiasinsurveyresults,itisdifficult,ifnotimpossible,toidentifyanacceptablelevelofresponse.Nonetheless,itiswidelyheldthataresponserateof75%to90%issufficienttosupportgeneralization.Iftheresponserateforasurveyismorethan90%,researchersareprobablyjustifiedingeneralizingfromthesampletothepopulation.Iftheresponseratefallsbetween75%and90%,generalizationmaybejustified,particularlyifotherconditionsidentifiedinthissectionarealsomet.Whensurveyresponseratesfallbelow75%,straightforwardgeneralizationsfromsampletopopulationaretenuousatbest,andthedatashouldprobablybesubjectedtosomeoftheotherproceduresoutlinedinthissectiontoassessandpossiblycorrectforbiascausedbynonresponse.

Third,thetypeofresearchisimportant.Nonresponseislessseriousforpreliminaryresearchorresearchdesignedtosupporttheorydevelopment—inpartbecausesuchstudiesarerarelyinterpretedalone.Inaddition,manystudiesofthistypefocusonrelationshipsbetweenmultiplevariables(discussedlater)thatmaybelessinfluencedbynonresponse.Finally,thereissomeevidencethattheimpactofnonresponseisgreaterfortheestimationofunivariatepopulationparameters(e.g.,averageageoflibraryusers)thanfortheestimationofbivariatecorrelations(e.g.,relationshipbetweenageandlibraryresourceuse),multivariaterelationships(e.g.,facultystatus,salary,andlibrarysizeaspredictorsoflibrarianjobsatisfaction),ortestsofdifferencesbetweengroupswithinthelargersample(e.g.,comparisonoftheeffectivenessofbibliographicinstructionforface-to-faceversusdistancelearners).Caremustbetaken,however,intheapplicationofthiscriterion.Althoughasmallerimpactofnonresponseonbivariateandunivariaterelationshipshasbeenobservedinempiricaldata(Goodman&Blum,1996;vanGoor&Verhage,1999)andthereissometheoreticalbasistoassumethattheobservationiscorrect(Alexander,Barrett,Alliger,&Carson,1986),thisconclusionisnotunchallenged.Brehm(1993)presentedconvincingtheoreticalevidenceandsupportingMonteCarlosimulationdatademonstratingthatnonresponsecanhavealargebiasingeffectonmultivariaterelationshipsiftheselectioncriteriaandthedependentvariableofinterestarerelated.

3.3.Limitingsurveyconclusions

Thepreviousdiscussionindicatesthat,inmanycasesifnotmost,itisinappropriatesimplytoignorenonresponse.Analternativestrategyistolimitsurveyconclusionscommensuratewiththelevelofnonresponse.Ifthesurveyresponserateislessthan75%,theresearchershould,ataminimum,ensurethatthereaderisawareofthelimitationtothegeneralizabilityofsurveyresultsarisingfromnonresponse.Abetterapproachmaybetoassumethatnonresponsehasintroducedbiasinsurveyresults;itisthenpossibletoestimatethelikelyimpactofthatbiasusingstatisticaltechniques.Thesetechniquesworkbycalculatinghow

J.Burkell/Library&InformationScienceResearch25(2003)239–263247

nonrespondentsmighthaveansweredusingworstcaseoraveragecaseassumptions;researchersthenusethisinformationtoreducetheprecisionwithwhichresultsarestated.Thereareproceduresavailableforcalculatingthemaximumamountofbiasintroducedbynonresponseinaunivariatepopulationestimate,suchastheestimateofaveragejobsatisfactionamonglibrarians(Alexanderetal.,1984;Chen,1996;Cochran,1963).Somesurveyanalysesincludesubgroupcomparisons,suchasanexaminationofthecomplianceofsmall,medium,andlargepubliclibrarieswithrecommendationsregardingInternetconnec-tivity.Whensubgroupdifferencesaretestedusingttestsoranalysesofvariance,itispossibletoestimatethedegreetowhichsuchdifferencesareaffectedbynonresponse(Viswesvaranetal.,1993).Thesetechniqueseffectivelyreducetheprecisionofsurveyresultstocompensateforsamplingbiasfromnonresponse.Unlessresponseratesareveryhigh,however,theytendtobeoverlyconservative,andresearcherswhousetheseapproachesarelikelytounderes-timatethesignificanceoftheirresults.Thus,thesestatisticsarenotwidelyusedbecausetheyworkbestwhentheyareneededleast:atlowlevelsofnonresponse.3.4.Assessingandcorrectingforsamplingbiascausedbynonresponse

Amorewidelyusedapproachtotheissueofnonresponseistoattempttodeterminewhethernonresponsehasintroducedsamplingbiasandtocorrectthatbiasifitisfound.Themostcommonlyusedstrategycomparesrespondentstoasecondgroupthateitherincludesorrepresentsnonrespondentsusingoneofthethreespecificstrategiespresentedinthissection.Qualitativecomparisons(e.g.,Clougherty,Forys,Lyles,Persson,Walters,&Washington-Hoagland,1998;Hart,1998)providesomeinsightintotheimpactofnonresponse,butabetterapproachistotestthestatisticalsignificanceofanyobserveddifferences.Asignificantdifferencebetweenrespondentsandthecomparisongroupsuggeststhepresenceofsamplingbias.Ifsuchadifferenceisfound,researcherscanattempttocorrectthebiasbyreweightingthedatatoaccountfortheunderrepresentationofsubgroupsresultingfromthenonresponse(seesection3.5).

Therespondentscanbecomparedwiththepopulationfromwhichtheyweredrawninanattempttoanswerthefollowingquestion:Dorespondentsrepresentarandomsampleofthepopulation?Theapplicationofthisapproachislimitedtocomparisonsforwhichdataareavailablebothforrespondentsandforthepopulationasawhole—generallydemographicdescriptors.Astatisticaltestisappliedtothesedatatodeterminetheprobabilitythatthesampleofrespondentsisdrawnfromthepopulation.Theappropriatestatisticaltestdependsonthelevelofmeasurementofvariablebeingcompared(intervalorratioversusordinalornominal)and,forintervalorratiovariables,whatisknownaboutthepopulationdistribution(Table1).

Asecondapproachisbasedonresearchindicatingthatnonrespondentsaremorelikelaterespondersthanearlyrespondersinbothexpressedattitudesanddemographics(Dalecki,Whitehead,&Blomquist,1993;Green,1991;Pearl&Fairley,1985).Thisapproachdividesthegroupofrespondentsintoatleasttwosubgroups:earlyrespondentsandlate(orreluctant)respondents.Thedivisioncanbebasedonfactorssuchasthetimeelapsedbetweeninitialmailoutandresponseorthenumberofremindersrequiredbeforeresponse.Thechoiceof

248J.Burkell/Library&InformationScienceResearch25(2003)239–263

Table1

StatisticstotestwhetherthesampleisdrawnfromthepopulationLevelof

measurement,variabletobecomparedIntervalorratioIntervalorratio

InformationknownaboutthepopulationMeanandSDMean

Additionalconsiderations

AppropriatestatisticZscore

Onesamplettest

Reference

Howell

(1997,p.181–183)Howell

(1997,p.183–191)Howell

(1997,p.146–149)

OrdinalornominalProportionfallingNotedistinctionbetweenChi-square

Goodness-of-Fittest,intothedifferentthetreatmentof

one-wayclassificationvariableswithonlygroups

2categoriesandthosewith>2categories

statistictocomparethegroupsdependsonthenumberofgroupsidentifiedandthelevelofmeasurementofthevariableofinterest(Table2).

Athirdapproachcomparesrespondentswithnonrespondentsonvariablesthatareavailableforbothgroups.Ifrespondentsfromtheoriginalsamplecanbeidentified(e.g.,throughnumerickeysonmailed-outsurveys),itispossiblealsotoidentifynonrespondents.Inthiscase,externaldatasourcescanprovideinformationaboutbothgroups(e.g.,administrativedatamightbeavailablefortheentiresample).CrawfordandRice(1997)usedthisstrategytoconcludethattherewasnosignificantbiasfromnonresponseintheirsurveyoftheeffectsofautomationonliberalartscollegelibraries.Alternatively,subjectscanexplicitlybeofferedtheoptiontorefusetocompletethesurvey,andsubjectswhorefusecanbeaskedtoanswerasmallnumberofquestions(Senf,1987).Thisstrategy,whichisaversionofthe‘‘door-in-the-face’’techniqueforinducingcompliance(Cialdinietal.,1975),mayallowasmallamountofdatatobecollectedfrompeoplewhodonotcompletetheentiresurvey.Table2identifiestheappropriatestatisticforthecomparisonsbetweenrespondentsandnonrespondents.

Thefirstandthirdapproachesoutlinedpreviouslyarelimitedbythedataavailableforthetwogroupstobecompared(usuallythisisrestrictedtodemographicinformation).Thus,with

Table2

StatisticstotestfordifferencebetweenrespondentsandnonrespondentsNo.levels,independentorcomparisonvariablez22>2

Levelofmeasurement,variableofinterestOrdinalornominalIntervalorratioIntervalorratio

AppropriatestatisticChi-square

IndependentsamplesttestBetweengroupsanalysisofvariance

Reference

Vaughan(2001,p.75–88)Howell(1997,p.149–159)Vaughan(2001,p.111–122)Howell(1997,p.198–206)Vaughan(2001,p.125–138)Howell(1997,p.319–367)

J.Burkell/Library&InformationScienceResearch25(2003)239–263249

theseapproaches,itisonlypossibletoassesssamplingbiaswithrespecttothesevariables,andnotwithrespecttothesubstantivesurveydata.Thesecondapproachallowsdirectcomparisonofsurveyresponses,butitisbasedonthe(possiblyincorrect)assumptionthatnonrespondentscanbeadequatelyrepresentedbyreluctantrespondents.Abetterwaytoassesstheimpactofsamplingbiasistocomparedirectlythesurveydataforrespondentsandnonrespondents(Fuller,1974).Deming(1960)suggestedamethodtoaccomplishthis:selectarandomsubsetofnonrespondentsandpursue100%responseratesfromtheseindividuals.Thedatacollectedfromtheseconvertednonrespondentscanbecomparedwithdatafromtheinitiallyrespondingsampletoidentifythedegreeofbiasfromnonresponseonanyvariablemeasuredbythesurvey.Thislaststrategymayrepresentthebestwaytoaddresstheimpactofnonresponseonsurveydata;however,therearetwofactorsthatresearchersshouldconsider.First,gatheringdatafromasampleofinitialnonrespondentsislikelytobetime-consumingandexpensive.Second,careshouldbetakenthattheprocessofgatheringthedatafromtheseinitialnonrespondentshasminimalimpactontheresponsesthemselves.Forexample,theresearchershouldbeawarethatlackoftimemayhavepreventedtheseindividualsfromresponding.Ifso,theresearchermusttakespecialcaretofindanopportunityfordatagatheringwhentherespondenthassufficienttimetodevotetotheprocess;otherwise,thedatacollectedcouldbeofquestionablequality.Ultimately,ifthisprocedureisfollowed,theresearchermaychoosetolimitthedatacollectedtoonlythemostimportantsurveyquestionstominimizeboththecostsofcollectingthedataandtheimpositiononthealreadyreluctantrespondent.3.5.Adjustingforsamplingbiascausedbynonresponse

Whensamplingbiasisidentifiedusingthemethodsoutlinedpreviously,reweightingofobtaineddataisthestrategymostfrequentlyusedtocompensateforsamplingbiascausedbynonresponse.Usingastatisticalpackage,suchasSPSS(SPSS,Inc,Chicago,IL),selectedcasescanbegivenadditionalweightinanalyses.Theweightingfactorsarechosentoremedythedifferencebetweensampleproportionsandpopulationproportionsforvarioussubgroups.Forexample,ifthepopulationissplitevenlybetweenmenandwomen,but60%oftherespondentsaremen,weightscanbeappliedtocompensateforthedifference.Theweightappliedtoeachrespondentiscalculatedasthepopulationproportiondividedbythesampleproportion.Inthiscase,theweightformalerespondentswouldbe50%dividedby60%,or0.8333.Theweightappliedtofemalerespondentswouldbe50%dividedby40%,or1.25.Whenstatisticssuchasaveragesarecalculatedovertheweighteddata,itisasifthesamplewassplit50/50intermsofgender.Holt,Elliott,andMoore(1999),usedanalternative,mathematicallyequivalentmethodforcalculatingweights.Theycalculatedtheweightofeachcaseasthenumberintheweightingclassfortheentirepopulationdividedbythenumberintheweightingclassforthesample.Foramoredetaileddiscussionofweightingandotherstatisticaltechniquestocompensatefornonresponse,seeGroves(1989),Grovesetal.(2001),LittleandRubin(1987),orKalton(1983).

Althoughweightingiswidelyusedtocompensatefornonresponse,weightingadjustmentsarenotalwayseffectiveinreducingbias(Brehm,1993;vanGoor&Stuiver,1998),and,insomecases,theymayevenexacerbatetheproblem(Brehm,1993).Essentially,theprocessof

250J.Burkell/Library&InformationScienceResearch25(2003)239–263

weightingreplicatesrespondentswithineachweightingclasstocompensateforthosewithintheclasswhodidnotrespond.Thus,weightingisbasedontheassumptionthat,withineachweightingclass,therearenosystematicdifferencesbetweenrespondentsandnonrespondents(Mandell,1974).Ifthisassumptionisnotmet(andwithoutactuallycollectingdatafromnonrespondents,thereisnowaytodeterminewhetheritis),weightingcannotcompensateforthesamplingbiasintroducedbynonresponse.Morecomplexmodelingapproaches(Achen,1986;Glynn,Laird,&Rubin,1986;Heckman,1976,1979;Heckman&Robb,1986)arelesssusceptibletothiscriticism,althoughtheseapproachesrequirethattheresearchermakeassumptionsabouttherelationshipbetweenthelikelihoodofparticipationandthevariablesofinterest(Brehm,1993;Wainer,1989).Deming(1960)suggestedtheonlyapproachthatdoesnotrequireassumptionsaboutthenatureofeitherthemechanismsthatresultinnonresponseorthedatathatwouldhavebeenobtainedfromnonrespondents.Inthatapproach,thedatafromtheconvertednonrespondentsrepresentthedatafromtheentiregroupthatdidnotrespond.Iftheresearcherissuccessfulincollectingdatafromarandomsubsetoftheoriginalnonrespondents,appropriateweightingofthedataprovidedbythisgroupadequatelycorrectsfornonresponsebias.

4.NonresponseintheLISliterature

Summariesofsurveyspublishedinavariousdisciplinesindicatethat,althoughresponseratesarelowenoughtoconstituteanissuefortheinterpretationofresults,alargeproportionofpublishedarticlesfailtorespondtotheissue(Cummingsetal.,2001;Lindner,Murphy,&Briers,2001).ThissectionattemptstodeterminewhetherasimilarsituationexistsintheLISliterature.Specifically,anempiricalstudyofsurveyspublishedinLISisusedtoexaminethefollowingquestions:

??????

WhatistheextentofnonresponseinsurveyspublishedintheLISliteratureandhowseriousisthislevelofnonresponsefortheinterpretationofsurveyresults?Whatmethodologicalstrategiesdoresearcherstaketoreducenonresponse?

Doresearchersattempttodeterminethedegreeofand/orstatisticallyaccountfornonresponsebias,andifso,how?

Conclusionsarebasedonacensusofsurveyspublishedfrom1996through2001inthreeLISjournals:PublicLibraries,College&ResearchLibraries,andLibrary&InformationScienceResearch.Theyears1996through2001representthemostrecentsixfullyearsofpublicationforthesejournalsatthetimethecensuswascompleted.Thisconstitutesapurposivesample,chosentocoverabroadrangeofrecentLISresearchinbothacademicandpubliclibrarysettings.ThesurveysexaminedarenotarandomsampleofsurveyspublishedinLIS,andthereforecaremustbetakeningeneralizingbeyondthespecificresults.Thepurpose,however,isonlytodeterminewhethernonresponseandthetreatmentofitareaproblemintheLISliteratureandnottodrawquantitativeconclusionsaboutthedegreeoftheproblem.Thepurposivesampledescribedissufficient,ifnotideal,tomeetthoseneeds.

J.Burkell/Library&InformationScienceResearch25(2003)239–263251

Thesurveysselectedmetthefollowingcriteria:

??????

Thegoalofthesurveywastodescribethepopulationofinterest(thus,surveyswerenotincludediftherewasnointentiontoapplyresultstothepopulation).

Eithertheentirepopulationwasselectedforparticipation(acensus)ortheselectedparticipantsrepresentedarandomsampleofthepopulation.Theresponseratewasreported.

TheAppendixpresentsthelistofarticlesthatreportedthesurveysusedintheanalysis.Fromthepublishedreportofeachsurvey,informationwasextractedregardingtheresponserate,themethodologicalstrategiesusedtoreducenonresponse,andanyattemptstoassessand/orcorrectforbiascausedbynonresponse(seeTable3fordetails).Inmanyofthereportsexamined,themethodologicaldetailsaresketchy(thisisconsistentwiththeresultsofMcKechnieetal.,2002).Forthepurposesofthisanalysis,itisassumedthatmethodologicalvariationsarenotusedunlesstheyareexplicitlymentionedinthereport.Anybiasthatarisesfromthisassumptionwillleadtotheunderestimationoftheuseofthestrategies.Detailssuchasthese,however,shouldbeincludedinthemethodsdescriptionsofpublishedstudies,andthereforetheassumptionisdeemedappropriate.

Table3

RecordedsurveyinformationVariableResponserateGroupsurveyed

LevelofmeasurementRatioNominal

Categories

—

LibraryrepresentativesLibrariansUsers

UnivariatedescriptionMultivariatedescriptionModeldevelopment/testingFace-to-faceTelephoneMailE-mailOtherYesNoYesNoYesNoYesNoNone

Comparisononly

Comparisonandcorrection

TypeofconclusionNominal

MethodofadministrationNominal

PersonalizationofsurveymaterialsRemindersIncentives

Self-addressed,stampedenvelopeNonresponsebiasassessment/correction

NominalNominalNominalNominalNominal

252J.Burkell/Library&InformationScienceResearch25(2003)239–263

4.1.Results

Duringtheyears1996through2001,76surveysreportedin72articles(onearticleincludedthreesurveys,andtwoothersincludedtwosurveyseach)metthecriteriaforinclusion.Libraryrepresentatives(usuallythelibrarydirector)werethemostfrequentlysurveyedgroup(32surveys,42.1%).Librariansrespondingasindividualsandnotrepre-sentativesofthelibrarywerethetargetpopulationin21surveys(27.6%),whereasusers(e.g.,faculty,students,medicalpatients)madeupthepopulationfor23surveys(30.3%).4.1.1.SurveynonresponseintheLISliterature

Theaverageresponserateacrossallsurveysexaminedwas62.9%(SD=19.6),and21surveys(27.6%)hadresponseratesof75%orgreater.Thedataarebrokendownbyjournal,year,populationtype,andresearchgoal(Tables456–7).Inferentialtestsfortheimpactofthesevariablesonresponserateareinappropriategiventhatthedatarepresentapopulationcensus;however,effectsizes,indicatingtheproportionofresponseratevariabilitythatisaccountedforbyeachindependentvariable,canbecalculated.Effectsizeswerelessthan0.1(<10%ofvarianceaccountedfor)forjournal,year,andresearchgoal,suggestingthatthesevariableshadlittleimpactonresponserate.Populationtype,however,accountedformorethan25%ofthevarianceinresponserate(effectsizeof0.256),withlibraryrepresentativesshowingthehighestresponserates(72%),followedbylibrarians(65%)andothergroups(49%).

NonresponseisanissueinLISresearch.Althoughtheaverageresponseratewasrelativelyhigh,approximatelyonethirdofthoseidentifiedforsurveyparticipationdidnotprovidedata.Furthermore,almostthreefourthsofthesurveysexaminedhadresponseratesbelowthatgenerallyrequiredforgeneralizability.

4.1.2.Useofmethodologicalvariationstoreducenonresponse

Methoddescriptionsforeachsurveywereexaminedtodeterminedetailsaboutthesurveyadministration.Mostofthesurveys(66,or86.8%)wereadministeredbymail,withfivesurveys(6.6%)administeredbye-mail,three(3.9%)administeredbytelephone,andtheremainingtwosurveys(2.6%)administeredusingothermeans(e.g.,handdeliveryor

Table4

Resultsbyjournal

No.surveys(%oftotal)45(59.2)17(22.4)

Averageresponserate,%(SD)62.4(18.5)67.5(16.7)

No.surveyswithresponserates>75%(%withingroup)9(20)7(41)

Journal

College&

ResearchLibrariesLibrary&

InformationScienceResearch

PublicLibraries

14(18.4)58.8(22.2)5(36)

J.Burkell/Library&InformationScienceResearch25(2003)239–263

Table5

Resultsbyyear

No.surveys(%oftotal)19(25)11(14.5)13(17.1)17(22.4)11(14.5)5(6.6)

Averageresponserate,%(SD)696266536368(10.1)(20.2)(15.4)(26.4)(13.9)(31.8)

253

Year199619971998199920002001

No.surveyswithresponserates>75%(%withingroup)534423(26.3)(27.3)(30.8)(25.5)(18.2)(60.0)

combinationofmethods).Mostsurveysweredeliveredtoaspecific,namedindividual(67,or88.2%),andinalmosthalfofthesurveys(37,or48.7%),atleastonereminderwassenttosurveyrespondents.Only10ofthesurveys(15.2%ofthemailedsurveys)providedrespondentswithaself-addressed,stampedenvelopeforsurveyreturn,andfour(3.9%)sentprenotificationofthesurveytopotentialrespondents.Onlysixsurveys(7.9%)usedincentives.Althoughprepaidmonetaryincentiveshavebeendemonstratedtobemosteffective,allincentiveswerenonmonetary,andatleastfivewereofferedforsurveycompletionratherthanbeingincludedwiththesurveywhenitwasdelivered(itwasimpossibletotellfromthemethoddescriptionofthesixthsurveywhethertheincentivewasofferedinitiallyorforcompletiononly).Thus,thedatashowthatmanymethodolog-icalvariationsproventoincreaseresponserateswerenotwidelyusedinthesurveysstudied,andthosevariationsthatwereusedwerenotalwaysusedinthemosteffectivemannerpossible.

4.1.3.Assessmentofandcorrectionforresponsebias

Ninesurveys(11.8%)explicitlyaddressedtheissueofnonresponsebeyondsimplyacknowledgingresponserate.Oneassessedthebiasarisingfromnonresponseandcorrectedbyreweightingtheresults.Anotherpresentedtheresultsofstatisticalcomparisonsofrespondentswiththepopulationofinterest,butnosignificantdifferenceswereidentifiedandthereforenocorrectionwasattempted.Finally,sevenreportspresentedeitherpopulationparametersorcomparativedataforrespondentsandnonrespondentsbutdidnottestfor

Table6

Resultsbypopulationtype

No.surveys(%oftotal)32(42.1)21(27.6)23(20.3)

Average

responserate,%(SD)72(11.5)65(14.9)49(24.2)

No.surveyswithresponserates>75%(%withingroup)11(34)6(29)4(17)

PopulationtypeLibraryrepresentativesLibrariansUsers

254J.Burkell/Library&InformationScienceResearch25(2003)239–263

Table7

Resultsbyresearchgoal

No.surveys(%oftotal)30(39.5)37(48.7)9(11.8)

Averageresponserate,%(SD)59.1(22.2)65.1(16.6)66.3(19.6)

No.surveyswithresponserate>75%(%withingroup)8(26.7)9(24.3)4(44.4)

Researchgoal

UnivariatedescriptionMultivariatedescription

Modeltestingordevelopment

significantdifferencesbetweenthegroups.Thelargemajorityofsurveysexamined,therefore,donotaddresstheissueofnonresponsebeyondreportingresponserate.4.2.Discussion

Theresultsofthisoverview,althoughnotgeneralizabletoallsurveyspublishedintheLISliterature,suggestthatLISresearchersarenotimmunetotheproblemofnonresponse.Theaverageresponserateforthesurveysidentifiedinthethreejournalsstudiedwas63%,avaluethatiscomparabletotheaverageresponserateof61%observedforsurveysofphysiciansovertheyears1986to1995(Cummingsetal.,2001).Althoughtheaveragerateofresponseisrelativelyhigh,almostthreequartersofthesurveysexaminedhadresponserateslessthan75%,alevelthatistraditionallyheldtoberequiredforgeneralizationfromsampletopopulation.Furthermore,mostsurveysinLISareconductedtodescribeapopulation,ratherthantotestordevelopamodel,whichissignificantbecausethesetypesofresultswillbemostaffectedbynonresponse.

Thehighestaverageresponserateswereobservedforsurveysoflibraryrepresentatives,followedbythoseoflibrarians,andfinally,surveysofusergroups(e.g.,faculty).Thisorderingofresponseratesmaybeexplainedbythehigherresponseratesgenerallyobservedforsurveysthatareofintrinsicinteresttothosecompletingthem(Donald1960;Martin,1994;Senf,1987).BecausethegeneralfocusofresearchpublishedinLISislibrariesandinformationservices,itseemslikelythatlibrariesandlibrarianswouldbemoreinterestedinthetopicsthanwouldbemembersofusergroups.

ItappearsthatLISresearcherscoulddoabetterjobofminimizingnonresponseandimplementingstrategiestoimprovetheinterpretationofsurveyresultsinthecontextofnonresponse.Thereislittleutilizationofmethodologicalvariationsthathavebeendemon-stratedtoimproveresponserates.Almostallsurveysreportedinthesethreejournalsovertheyears1996through2001usedtraditionalmailore-mailasthemethodofsurveydelivery.Fewstudiesusedprenotification,reminders,self-addressed,stampedenvelopeforsurveyreturns,orincentivestoencouragehighresponserates.Furthermore,thosestudiesthatprovidedincentivesimplementedtheminarelativelyineffectivemanner:theygenerallyofferednonmonetaryincentivesforcompletedsurveysratherthanincludingasmallmonetaryincentivewiththeinitialsurveydelivery.Onlyasmallproportionofstudiesacknowledgedthepossibilityofsamplingbiascausedbynonresponseandattemptedtoassessorcorrectforthatbias.

J.Burkell/Library&InformationScienceResearch25(2003)239–263255

5.Conclusion

Nonresponsepresentsavirtuallyunresolvabledilemma:toknowwhethernonresponsehasbiasedsurveyresults,researchersmustknow(orguessat)thedatathatnonrespondentswouldhaveprovided.Thestrategiesprovidedinthisarticledonoteliminatethedilemma.Theymerelyrepresentabestpossibleresponsetoanimpossibleproblem.Thatbeingsaid,itisevidentthatsurveynonresponseisanissueinLIS,asitisinotherdisciplines,andLISresearchersneedstrategiesforminimizingtheproblemofnonresponseandtheimpactofnonresponseonsurveyconclusions.Toachievethesegoals,thefollowingstrategiesaresuggested:

????

??

??

First,researchersshouldconcentrateonreducingnonresponsebyusingmethodologicalvariationsdemonstratedtoincreaseresponserates.

Ifnonresponseremainshigh(>25%)despitetheseefforts,caremustbetakentolimitsurveyconclusionsappropriately,giventhepossibilityofsamplingbiascausedbynonresponse.

Wheneverpossible,researchersshouldattempttoassessthedegreeofbiasintroducedbynonresponse.Thebestwaytoaccomplishthismaybebyusingadditionaleffortandresourcestocollectdatafromarandomsampleoftheinitialnonrespondentsandcomparingdatafrominitialrespondentstotheresultsgatheredfromthisgroup.

Finally,insomecases(whengeneralizabilityisofparticularimportanceandtherequireddataareavailable),researcherscanconsiderstatisticalprocedures,includingsamplereweighting,thatmayhelptoreducetheimpactofnonresponseonresults.

Thesestrategiesprovideaprincipledplanforfillingintheblindspotthatarisesfromsurveynonresponse.

Appendix.Articleswithsurveysmeetinginclusioncriteria

1.Immroth,B.F.,&Lance,K.C.(1996).Outputmeasuresforchildren’sservicesinpubliclibraries.PublicLibraries,35,240–245.

2.Chelton,M.K.(1997).Threeinfivepubliclibraryusersareyouth:ImplicationsofsurveyresultsfromtheNationalCenterforEducationStatistics.PublicLibraries,36,104–108.

3.Johnson,C.P.(1998).LibraryemployeeswithouttheALA-accreditedMaster’sdegree.PublicLibraries,37,40–46.

4.Baker,L.M.,Spang,L.,&Gogolowski,C.(1998).TheprovisionofconsumerhealthinformationbyMichiganPublicLibrarians.PublicLibraries,37,250–255.

5.Holt,G.E.,Elliott,D.,&Moore,A.(1999).Placingavalueonpubliclibraryservices.PublicLibraries,38,98–108.

256J.Burkell/Library&InformationScienceResearch25(2003)239–263

6.Gregory,G.(1999).TheLibraryServicesandTechnologyAct:HowchangesfromLSCAareaffectinglibraries.PublicLibraries,38,378–382.

7.Liu,Y.Q.,&Zweizig,D.(2000).Publiclibraryuseofstatistics:Asurveyreport.PublicLibraries,39,98–105.

8.Crawford,G.A.(2000).Internetissues:Access,filtering,andasurveyofPennsylvaniapubliclibraries.PublicLibraries,39,162–167.

9.Gilton,D.G.(2000).Onwardandupward:Promotionofpubliclibrarians.PublicLibraries,39,214–219.

10.ThePublicLibrariesCommitteeoftheNewEnglandMusicLibraryAssociation.(2000).MusiccollectionandacquisitionpracticesinConnecticutpubliclibraries.PublicLibraries,39,348–355.

11.Staples,S.(2001).Fastandslowlanesoftheinformationhighway:Librariandecisionmakingaboutelectronicinformationservices.PublicLibraries,40,294–297.

12.Karp,R.,&Keck,A.(1996).Theologicallibrarianship:Towardaprofileofaprofession.College&ResearchLibraries,57,35–42.

13.Farr,K.W.,&Scott,R.N.(1996).EarningsdeterminantsoflibraryfacultyintheUniversitySystemofGeorgia.College&ResearchLibraries,57,77–87.

14.Allen,F.R.(1996).Materialsbudgetsintheelectronicage:Asurveyofacademiclibraries.College&ResearchLibraries,57,133–143.

15.Hurd,J.M.(1996).ARLacademicscienceandtechnologylibraries:Reportofasurvey.College&ResearchLibraries,57,144–160.

16.Hseih-Yee,I.(1996).Studentuseofonlinecatalogsandotherinformationchannels.College&ResearchLibraries,57,161–175.

17.Metz,P.,&Stemmer,J.(1996).Areputationalstudyofacademicpublishers.College&ResearchLibraries,57,234–248.

18.Dilmore,D.H.(1996).Librarian/facultyinteractionatnineNewEnglandcolleges.College&ResearchLibraries,57,274–284.

19.Koenig,M.E.,Morrison,R.,&Roberts,L.(1996).Facultystatusforlibraryprofessionals:Itseffectonjobturnoverandjobsatisfactionamonguniversityresearchlibrarydirectors.College&ResearchLibraries,57,295–300.

20.Patterson,L.,&Taylor,R.H.(1996).Triballycontrolledcommunitycollegelibraries:Aparadigmforsurvival.College&ResearchLibraries,57,316–329.

21.Via,B.J.(1996).Publishinginthejournalliteratureoflibraryandinformationscience:Asurveyofmanuscriptreviewprocessesandacceptance.College&ResearchLibraries,57,365–376.

22.Carrigan,D.P.(1996).Data-guidedcollectiondevelopment:Apromiseunfulfilled.College&ResearchLibraries,57,429–437.

23.Smith,N.,&Tibbo,H.R.(1996).Librariesandthecreationofelectronictextsforthehumanities.College&ResearchLibraries,57,535–553.

24.Carpenter,S.(1996).TheAmericanswithDisabilitiesAct:AccommodationinOhio.College&ResearchLibraries,57,555–566.

25.Thornton,J.K.(1997).CarpaltunnelsyndromeinARLlibraries.College&ResearchLibraries,58,9–18.

J.Burkell/Library&InformationScienceResearch25(2003)239–263257

26.Leckie,G.L.,&Brett,J.(1997).JobsatisfactionofCanadianuniversitylibrarians:Anationalsurvey.College&ResearchLibraries,58,31–47.

27.Coleman,V.,Xiao,Y.D.,Bair,L.,&Chollett,B.(1997).TowardaTQMparadigm:UsingSERVQUALtomeasurelibraryservicequality.College&ResearchLibraries,58,237–251.

28.Baldwin,V.A.,Gibbs,W.J.,&Slough,M.M.(1997).Initiatinganeffectivepersonneldevelopmentprogram.College&ResearchLibraries,58,267–277.

29.Kirkland,J.J.(1997).Themissingwomenlibrarydirectors:Deprivationversusmentoring.College&ResearchLibraries,58,376–384.

30.Libby,K.A.,&Caudle,D.M.(1997).Asurveyontheoutsourcingofcataloginginacademiclibraries.College&ResearchLibraries,58,550–560.

31.HassardWilkins,J.L.,&Leckie,G.J.(1997).Universityprofessionalandmanagerialstaff:Informationneedsandseeking.College&ResearchLibraries,58,561–574.

32.Kirkpatrick,T.E.(1998).ThetrainingofacademiclibrarystaffoninformationtechnologywithinthelibrariesoftheMinnesotaStateCollegesandUniversitiessystem.College&ResearchLibraries,59,51–59.

33.O’Keefe,J.(1998).Smallcollegelibrarydirectors:Gettinginthedoorandsurvivingonthejob.College&ResearchLibraries,59,140–153.

34.Andaleeb,S.S.,&Simmonds,P.L.(1998).Explainingusersatisfactionwithacademiclibraries:Strategicimplications.College&ResearchLibraries,59,156–167.

35.Buttlar,L.,&Garcha,R.(1998).Catalogersinacademiclibraries:Theirevolvingandexpandingroles.College&ResearchLibraries,59,311–321.

36.Bell,S.J.,&Cronin-Kardon,C.(1998).Makingthelibrarymanagementsystemsacquisition:Achievingresolutionofatoughdecision.College&ResearchLibraries,59,348–361.

37.Preston,C.(1998).Perceptionsofdiscriminatorypracticesandattitudes:AsurveyofAfricanAmericanlibrarians.College&ResearchLibraries,59,434–445.

38.Leyson,H.M.,&Black,W.K.(1998).PeerreviewinCarnegieresearchlibraries.College&ResearchLibraries,59,512–522.

39.Clougherty,L.,Forys,J.,Lyles,T.,Persson,D.,Walters,C.,&Washington-Hoagland,C.(1998).TheUniversityofIowaLibraries’undergraduateuserneedsassessment.College&ResearchLibraries,59,572–584.

40.Leckie,G.,&Fullerton,A.(1999).Informationliteracyinscienceandengineeringundergraduateeducation:Facultyattitudesandpedagogicalpractices.College&ResearchLibraries,60,9–29.

41.Mitchell,B.,&Reichel,M.(1999).Publishorperish:Adilemmaforacademiclibrarians?College&ResearchLibraries,60,232–243.

42.Neely,T.Y.,&Winston,M.D.(1999).SnowbirdLeadershipInstitute:Leadershipdevelopmentintheprofession.College&ResearchLibraries,60,412–425.

43.Brown,C.(1999).Informationliteracyofphysicalsciencegraduatestudentsintheinformationage.College&ResearchLibraries,60,426–438.

44.Hart,R.L.(1999).Scholarlypublicationbyuniversitylibrarians:AstudyatPennState.College&ResearchLibraries,60,454–462.

258J.Burkell/Library&InformationScienceResearch25(2003)239–263

45.Lawson,K.G.,&Pelzer,N.L.(1999).Assessingtechnology-basedprojectsforpromotionand/ortenureinARLacademiclibraries.College&ResearchLibraries,60,464–476.

46.Bordeianu,S.,&Seiser,V.(1999).ParaprofessionalcatalogersinARLlibraries.College&ResearchLibraries,60,532–540.

47.Haines,A.(1999).Librarians’personalWebpages:Ananalysis.College&ResearchLibraries,60,543–550.

48.Weingart,S.J.,&Anderson,J.A.(2000).Whenquestionsareanswers:Usingasurveytoachievefacultyawarenessofthelibrary’selectronicresources.College&ResearchLibraries,61,127–134.

49.Winston,M.D.,&Li,H.(2000).Managingdiversityinliberalartscollegelibraries.College&ResearchLibraries,61,205–215.

50.Thornton,J.K.(2000).JobsatisfactionoflibrariansofAfricandescentemployedinARLacademiclibraries.College&ResearchLibraries,61,217–232.

51.Tenopir,C.,&Read,E.(2000).Patternsofdatabaseuseinacademiclibraries.College&ResearchLibraries,61,234–246.

52.Julien,H.(2000).InformationliteracyinstructioninCanadianacademiclibraries:Longitudinaltrendsandinternationalcomparisions.College&ResearchLibraries,61,510–523.

53.Maughan,P.D.(2001).Assessinginformationliteracyamongundergraduates:AdiscussionoftheliteratureandtheUniversityofCalifornia-Berkeleyassessmentexperience.College&ResearchLibraries,62,71–85.

54.Herring,S.D.(2001).FacultyacceptanceoftheWorldWideWebforstudentresearch.College&ResearchLibraries,62,251–258.

55.Brewer,J.,&Winston,M.D.(2001).Programevaluationforinternship/residencyprogramsinacademicandresearchlibraries.College&ResearchLibraries,62,307–315.56.Baker,L.M.(1996).Astudyofthenatureofinformationneededbywomenwithmultiplesclerosis.Library&InformationScienceResearch,19,67–81.

57.Cullen,R.,&Calvery,P.(1996).NewZealanduniversitylibrarieseffectivenessproject:Dimensionsandconceptsoforganizationaleffectiveness.Library&InformationScienceResearch,18,99–119.

58.Affleck,M.A.(1996).Burnoutamongbibliographicinstructionlibrarians.Library&InformationScienceResearch,18,165–183.

59.Alafiatayo,B.O.,&Yip,Y.P.(1996).Referencetransactionandthenatureoftheprocessforgeneralreferenceassistance.Library&InformationScienceResearch,18,357–384.

60.Crawford,G.A.,&Rice,R.E.(1997).Technology,power,andstructure:Developingamodeloftheeffectsofautomationonliberalartscollegelibraries.Library&InformationScienceResearch,19,265–300.

61.Kaminer,N.(1997).ScholarsandtheuseoftheInternet.Library&InformationScienceResearch,19,329–345.

62.Cha,M.,&Pungitore,V.L.(1998).CompliancewithpubliclibrarystandardsintheStateofOhio.Library&InformationScienceResearch,20,69–98.

J.Burkell/Library&InformationScienceResearch25(2003)239–263259

63.Hart,R.(1998).TherelationshipsbetweenworkrolesandinformationgatheringofthefacultyatSUNY,CollegeatFredonia.Library&InformationScienceResearch,20,163–185.

64.Nilsen,K.(1998).SocialscienceresearchinCanadaandgovernmentinformationpolicy:TheStatisticsCanadaexample.Library&InformationScienceResearch,20,211–234.

65.Travica,B.(1999).Organizationalaspectsofthevirtuallibrary:Asurveyofacademiclibraries.Library&InformationScienceResearch,21,173–203.

66.Wicks,D.A.(1999).Theinformation-seekingbehaviorofpastoralclergy:Astudyoftheinteractionoftheirworkworldsandworkroles.Library&InformationScienceResearch,21,205–226.

67.Chu,C.M.(1999).Literarycriticsatworkandtheirinformationneeds:Aresearch-phasesmodel.Library&InformationScienceResearch,21,247–273.

68.Bertot,J.C.,&McClure,C.R.(1999).U.S.publiclibraryoutletInternetconnectivity:Progressandstrategies.Library&InformationScienceResearch,21,281–298.

69.Sierpe,E.(1999).JobsatisfactionamonglibrariansinEnglish-languageuniversitiesinQuebec.Library&InformationScienceResearch,21,479–499.

70.Curry,A.,&Curtis,A.(2000).ConnectingtotheInternet:ThechallengeforCanada’scountyandregionallibraries.Library&InformationScienceResearch,22,77–103.

71.Majid,S.,Anwar,M.A.,&Eisenschitz,T.S.(2000).InformationneedsandinformationseekingbehaviorofagriculturalscientistsinMalaysia.Library&InformationScienceResearch,22,145–163.

72.Meho,L.I.,&Haas,S.W.(2001).Information-seekingbehavioranduseofsocialsciencefacultystudyingstatelessnations:Acasestudy.Library&InformationScienceResearch,23,5–25.

References

Achen,C.H.(1986).Thestatisticalanalysisofquasi-experiments.Berkley,CA:UniversityofCaliforniaPress.Albaum,D.S.,Evangelista,F.,&Medina,N.(1998).Roleofresponsebehaviorinsurveyresearch:Across-nationalstudy.JournalofBusinessResearch,42,115–125.

Alexander,R.A.,Alliger,G.M.,&Hanges,P.J.(1984).Correctionforrangerestrictionwhenthepopulationvarianceisunknown.AppliedPsychologicalMeasurement,8,431–437.

Alexander,R.A.,Barrett,G.V.,Alliger,G.M.,&Carson,K.P.(1986).Towardsageneralmodelofnon-randomsamplingandtheimpactonpopulationcorrelation:GeneralizationsofBerkson’sFallacyandrestrictionofrange.BritishJournalofMathematicalandStatisticalPsychology,39,90–105.

AmericanAssociationforPublicOpinionResearch.(2000).Standarddefinitions:Finaldispositionsofcasecodesandoutcomeratesforsurveys.RetrievedMay15,2002,fromhttp://www.aapor.org/ethics/stddef.html#mail.Ary,D.,Jacobs,L.,&Razavieh,A.(1996).Introductiontoresearchineducation(5thed.).FortWorth,TX:Holt,Rinehart,andWinston.

Baldauf,A.,Reisinger,H.,&Moncrief,W.C.(1999).Examiningmotivationstorefuseinindustrialmailsurveys.JournaloftheMarketResearchSociety,41,345–353.

Berelson,B.(1949).Thelibrary’spublic:Areportofthepubliclibraryinquiry.NewYork:ColumbiaUni-versityPress.

260J.Burkell/Library&InformationScienceResearch25(2003)239–263

Blake,V.L.P.(1994).SinceShaughnessy:Researchmethodsinlibraryandinformationsciencedissertations,1975–1979.CollectionManagement,19,1–42.

Brehm,J.(1993).Thephantomrespondent:Opinionsurveysandpoliticalrepresentation.AnnArbor,MI:TheUniversityofMichiganPress.

Busha,C.H.,&Harter,S.P.(1980).Researchmethodsinlibrarianship:Techniquesandinterpretation.NewYork:AcademicPress.

Callison,D.(1997).Evolutionofmethodstomeasurestudentinformationuse.Library&InformationScienceResearch,19,347–357.

Chen,H.C.K.(1996).Direction,magnitude,andimplicationsofnonresponsebiasinmailsurveys.JournaloftheMarketResearchSociety,38,267–277.

Church,A.H.(1993).Estimatingtheeffectofincentivesonmailsurveyresponserates:Ameta-analysis.PublicOpinionQuarterly,57,62–79.

Cialdini,R.B.,Vincent,J.E.,Lewis,S.K.,Catalan,J.,Wheeler,D.,&Darby,B.L.(1975).Reciprocolconcessionsprocedureforinducingcompliance:Thedoor-in-the-facetechnique.JournalofPersonalityandSocialPsychology,31,206–215.

Clougherty,L.,Forys,J.,Lyles,T.,Persson,D.,Walters,C.,&Washington-Hoagland,C.(1998).TheUniversityofIowaLibraries’undergraduateuserneedsassessment.College&ResearchLibraries,59,572–584.

Cochran,W.G.(1963).Samplingtechniques(2nded.).NewYork:Wiley.

Cook,C.,Heath,F.,&Thompson,R.L.(2000).Ameta-analysisofresponseratesinweb-orInternet-basedsurveys.EducationalandPsychologicalMeasurement,60,821–836.

Cook,C.,&Thompson,B.(2001).Psychometricpropertiesofscoresfromtheweb-basedLibQUAL+studyofperceptionsoflibraryservicequality.LibraryTrends,49,585–604.

Cotton,C.C.,&Wonder,B.D.(1982).Mailsurveyresponserateandcorporatesize.PsychologicalReports,51,1218.

Couper,M.P.(2000).Websurveys:Areviewofissuesandapproaches.PublicOpinionQuarterly,64,464–494.Crawford,G.A.,&Rice,R.E.(1997).Technology,power,andstructure:Developingamodeloftheeffectsofautomationonliberalartscollegelibraries.Library&InformationScienceResearch,19,265–300.

Cummings,S.M.,Savitz,L.A.,&Konrad,T.R.(2001).Reportedresponseratestomailedphysicianques-tionnaires.HealthServicesResearch,35,1347–1355.

Dalecki,M.G.,Whitehead,J.C.,&Blomquist,G.C.(1993).Sampleofnonresponsebiasandaggregatebenefitsincontingentvaluation:Anexaminationofearly,late,andnonrespondents.JournalofEnvironmentalManagement,38,133–153.

Deming,W.E.(1960).Sampledesigninbusinessresearch.NewYork:Wiley.

Diamantopoulos,A.,&Schlegelmilch,B.B.(1996).Determinantsofindustrialmailsurveyresponse:Asurvey-on-surveysanalysisofresearchers’andmanagers’views.JournalofMarketingManagement,12,505–531.

Dillman,D.A.(2000).MailandInternetsurveys:Thetailoreddesignmethod.NewYork:Wiley.

Dimitroff,A.(1995).Researchforspeciallibraries:Aquantitativeanalysisoftheliterature.SpecialLibraries,86,256–264.

Donald,M.N.(1960).Implicationsofnonresponsefortheinterpretationofmailquestionnairedata.PublicOpinionQuarterly,24,99–114.

Feehan,P.E.,Gragg,L.W.,&Havener,M.W.(1987).Libraryandinformationscienceresearch:Ananalysisofthe1984journalliterature.Library&InformationScienceResearch,9,173–185.

Fidel,R.(1993).Qualitativemethodsininformationretrievalresearch.Library&InformationScienceResearch,15,219–247.

Fox,R.J.,Crask,M.R.,&Kim,J.(1988).Mailsurveyresponserate:Ameta-analysisofselectedtechniquesforinducingresponse.PublicOpinionQuarterly,52,467–491.

Fuller,C.H.(1974).Weightingtoadjustforsurveynonresponse.PublicOpinionQuarterly,38,239–246.Gajraj,A.M.,Faria,A.J.,&Dickinson,J.R.(1990).Acomparisonoftheeffectofpromisedandprovided

J.Burkell/Library&InformationScienceResearch25(2003)239–263261

lotteries,monetaryandgiftincentivesonmailsurveyresponserate,speed,andcost.JournaloftheMarketResearchSociety,32,141–162.

Gall,M.D.,Borg,W.R.,&Gall,J.P.(1996).Educationalresearch:Anintroduction(6thed.).WhitePlains,NY:Longman.

Glynn,R.J.,Laird,N.M.,&Rubin,D.B.(1986).Selectionmodelingversusmixturemodelingwithnon-ignorablenonresponse.InH.Wainer(Ed.),Drawinginferencesfromself-selectedsamples(pp.115–142).NewYork:Springer.

Goodall,D.(1996).‘‘Itain’twhatyoudo,it’sthewaythatyoudoit’’:Areviewofpubliclibraryresearchwithspecialreferencetomethodology.PublicLibraryJournal,11,69–76.

Goodman,J.S.,&Blum,T.C.(1996).Assessingthenon-randomsamplingeffectsofsubjectattritioninlongitudinalresearch.JournalofManagement,22,627–652.

Green,K.E.(1991).Reluctantrespondents:Differencesbetweenearly,late,andnonresponderstoamailsurvey.JournalofExperimentalEducation,59,268–276.

Green,K.E.(1996).Sociodemographicfactorsandmailsurveyresponse.PsychologyandMarketing,13,171–184.

Green,K.E.,Boser,J.A.,&Hutchinson,S.R.(1998).Response-ratedifferencesandresponse-enhancementeffectsbypopulationtype.PsychologicalReports,83,336–338.

Groves,R.M.(1989).Surveyerrorsandsurveycosts.NewYork:Wiley.

Groves,R.M.,Dillman,D.A.,Eltinge,J.L.,&Little,R.J.A.(2002).Surveynonresponse.NewYork:Wiley.Hart,R.L.(1998).TherelationshipsbetweenworkrolesandinformationgatheringofthefacultyatSUNYCollegeatFredonia.Library&InformationScienceResearch,20,163–185.

Heckman,J.F.(1979).Sampleselectionbiasasaspecificationerror.Econometrica,47,153–161.

Heckman,J.F.(1976).Thecommonstructureofstatisticalmodelsoftruncation,sampleselection,andlimiteddependentvariablesandasimpleestimatorforsuchmodels.AnnalsofEconomicandSoicialMeasurement,5,475–492.

Heckman,J.J.,&Robb,R.(1986).Alternativemethodsforsolvingtheproblemofselectionbiasinevaluatingtheimpactoftreatmentsonoutcomes.InH.Wainer(Ed.),Drawinginferencesfromself-selectedsamples(pp.63–107).NewYork:Springer.

Helgeson,J.G.,Voss,K.E.,&Terpening,W.D.(2002).Determinantofmail-surveyresponse:Surveydesignfactorsandrespondentfactors.PsychologyandMarketing,19,303–328.

Hernon,P.,&Schwartz,C.(2000).Surveyresearch:Atimeforintrospection.Library&InformationScienceResearch,22,117–122.

Holt,G.E.,Elliott,D.,&Moore,A.(1999).Placingavalueonpubliclibraryservices.PublicLibraries,38,98–108.

Hopkins,K.D.,&Gullickson,A.R.(1992).Responseratesinsurveyresearch:Ameta-analysisoftheeffectsofmonetarygratuities.JournalofExperimentalEducation,61,52–62.

Howell,D.C.(1997).Statisticalmethodsforpsychology(5thed.).Belmont,CA:DuxburyPress.

Hox,J.J.,&DeLeeuw,E.D.(1994).Acomparisonofnonresponseinmail,telephoneandface-to-facesurveys:Applyingmultilevelmodelingtometa-analysis.QualityandQuantity,28,329–344.

Jones,J.(1996).Theeffectsofnonresponseonstatisticalinference.JournalofHealthandSocialPolicy,8,49–62.

Julien,H.(1996).Acontentanalysisofrecentinformationneedsandusesliterature.Library&InformationScienceResearch,18,53–65.

Julien,H.,&Duggan,L.J.(2000).Alongitudinalanalysisoftheinformationneedsandusesliterature.Library&InformationScienceResearch,22,291–309.

Kaldenberg,D.O.,Koenig,H.F.,&Becker,B.W.(1994).Mailsurveyresponseratepatternsinapopulationoftheelderly:Doesresponsedeterioratewithage?PublicOpinionQuarterly,58,68–76.Kalton,G.(1983).Introductiontosurveysampling.BeverlyHills,CA:Sage.

Keeter,S.,Miller,C.,Kohut,A.,Groves,R.M.,&Presser,S.(2000).Consequencesofreducingnonresponseinanationaltelephonesurvey.PublicOpinionQuarterly,64,125–148.

262J.Burkell/Library&InformationScienceResearch25(2003)239–263

Kerlinger,F.N.(1986).FoundationsofBehavioralResearch.NewYork:Holt,Rinehart,andWinston.

Kettleson,M.J.(1995).Anassessmentoftheresponserateviathepostalserviceande-mail.HealthValues:TheJournalofHealthBehavior,EducationandPromotion,19,27–39.

Krosnick,J.A.(1999).Surveyresearch.AnnualReviewofPsychology,50,537–567.

Krysan,M.,Schuman,H.,Scott,L.,&Beatty,P.(1994).Responseratesandresponsecontentinmailversusface-to-facesurveys.PublicOpinionQuarterly,58,381–399.

Lakner,E.(1998).Optimizingsamplesforsurveysofpubliclibraries:Alternativesandcompromises.Library&InformationScienceResearch,20,321–342.

Lindner,J.R.,Murphy,T.H.,&Briers,G.E.(2001).Thehandlingofnonresponseinagriculturaleducation.Articlepresentedatthe28thAnnualNationalAgriculturalEducationResearchConference,NewOrleans,LA.RetrievedJuly8,2002,fromhttp://aaaeonline.ifas.ufl.edu/NAERC/2001/Papers/lindner1.pdf.Little,R.J.A.,&Rubin,D.B.(1987).Statisticalanalysiswithmissingdata.NewYork:Wiley.

Losee,R.M.,&Worley,K.A.(1993).Researchandevaluationforinformationprofessionals.SanDiego,CA:AcademicPress.

Mandell,L.(1974).Whentoweight:Determiningnonresponsebiasinsurveydata.PublicOpinionQuarterly,38,247–252.

Martin,C.L.(1994).Theimpactoftopicinterestonmailsurveyresponsebehaviour.JournaloftheMarketResearchSociety,36,327–338.

McKechnie,L.,Baker,L.,Greenwood,M.,&Julien,H.(2002).Methodologicaltrendsinhumanin-formationresearch.TheNewReviewofInformationBehaviourResearch:StudiesofInformationSeekinginContext,3.

Neuman,W.L.(2000).Socialresearchmethods(4thed.).NeedhamHeights,MA:AllynandBacon.

Palys,T.(1997).Researchdecisions:Quantitativeandqualitativeperspectives(2nded.).Toronto,ON:HarcourtBraceCanada.

Pearl,D.K.,&Fairley,D.(1985).Testingforthepotentialfornonresponsebiasinsamplesurveys.PublicOpinionQuarterly,49,553–560.

Perkins,G.H.,&Yuan,H.(2001).Acomparisonofweb-basedandpaper-and-pencillibrarysatisfactionsurveyresults.College&ResearchLibraries,62,369–377.

Powell,R.R.(1997).Basicresearchmethodsforlibrarians(3rded.).Greenwich,CT:Ablex.

Preitz,B.C.(1980–81).Themethodsoflibraryscienceresearch:Someresultsfromabibliographicsurvey.LibraryResearch,2,251–268.

Robinson,R.A.,&Agisim,P.(1951).Makingmailsurveysmorereliable.JournalofMarketing,15,415–424.Rogelburg,S.G.,Luong,A.,Sederbury,M.E.,&Cristol,D.S.(2000).Employeeattitudessurveys:Examiningtheattitudesofnoncompliantemployees.JournalofAppliedPsychology,85,284–293.

Roselle,A.,&Neufeld,S.(1998).Theutilityofelectronicmailfollow-upsforlibraryresearch.Library&InformationScienceResearch,20,153–161.

Senf,J.H.(1987).Theoptiontorefuse:Atoolinunderstandingnonresponseinmailedsurveys.EvaluationReview,11,775–781.

Sibbald,B.,Addington-Hall,J.M.,Brenneman,D.,&Freeling,P.(1994).Telephoneversuspostalsurveysofgeneralpractitioners:Methodologicalconsiderations.BritishJournalofGeneralPractitioners,44,297–300.

Simpson,C.W.(1992).Technicalservicesresearch,1988–1991.LibraryResources&TechnicalServices,36,383–408.

Singer,E.,VanHoewyk,J.,&Maher,M.P.(2000).Experimentswithincentivesintelephonesurveys.PublicOpinionQuarterly,64,171–178.

Sosdian,C.P.,&Sharp,L.M.(1980).Nonresponseinmailsurveys:Accessfailureorrespondentresistance.PublicOpinionQuarterly,44,396–402.

Thompson,B.(2000).Representativenessversusresponserate:Itain’ttheresponserate!PaperpresentedattheAssociationofResearchLibrariesMeasuringServiceQualitySymposiumontheNewCultureofAssessment:MeasuringServiceQuality,Washington,DC.

J.Burkell/Library&InformationScienceResearch25(2003)239–263263

Tomaskovic-Devey,D.,Leiter,J.,&Thompson,S.(1994).Organizationalsurveynonresponse.AdministrativeScienceQuarterly,39,439–457.

Tuckman,B.W.(1999).Conductingeducationalresearch(5thed.).FortWorth,TX:HarcourtBrace.

vanGoor,H.,&Stuiver,B.(1998).Awave-analysisofdistributionalbias,substantivebiasanddataqualityinamailsurveyamongDutchmunicipalities.ActaPolitica,33,179–196.

vanGoor,H.,&Verhage,A.L.(1999).Nonresponseandrecallerrorinastudyofabsencebecauseofillness:Ananalysisoftheireffectsondistributionsandrelationships.QualityandQuantity,33,411–428.

Vaughan,L.(2001).Statisticalmethodsfortheinformationprofessional:Apractical,painlessapproachtounderstanding,using,andinterpretingstatistics.Medford,NJ:InformationToday.

Viswesvaran,C.,Barrick,M.R.,&Ones,D.S.(1993).Howdefinitiveareconclusionsbasedonsurveydata:Estimatingrobustnesstononresponse.PersonnelPsychology,46,551–567.

Visser,P.S.,Krosnick,J.A.,Marquette,J.,&Curtin,M.(1996).Mailsurveysforelectionforecasting?AnevaluationoftheColumbusDispatchpoll.PublicOpinionQuarterly,60,181–227.

Wainer,H.(1989).Eelworms,bulletholes,andGeraldineFerraro:Someproblemswithstatisticaladjustmentandsomesolutions.JournalofEducationalStatistics,14,121–140.

Wayne,I.(1975–76).Nonresponse,samplesize,andtheallocationofresources.PublicOpinionQuarterly,39,557–562.

Yammarino,F.J.,Skinner,S.J.,&Childers,T.L.(1991).Understandingmailsurveyresponsebehavior:Ameta-analysis.PublicOpinionQuarterly,55,613–639.

本文来源：https://www.bwwdw.com/article/gq2x.html