Optical Engineering 4512, 127201 December 2006 Depth from motion and defocus blur Huei-Yung

One of the essential problems in computer vision is to recover the distance information of an object from captured images. Its application areas range from industrial inspection and reverse engineering to autonomous robot navigation

OpticalEngineering45 12 ,127201 December2006

Depthfrommotionanddefocusblur

Huei-YungLin,MEMBERSPIEChia-HongChang

NationalChungChengUniversityDepartmentofElectricalEngineering168UniversityRoad

Min-HsiungChia-Yi621,TaiwanE-mail:lin@u.edu.tw

Abstract.Findingthedistanceofanobjectinascenefromintensityimagesisanessentialprobleminmanyapplications.Inthiswork,wepresentanovelmethodfordepthrecoveryfromasinglemotionanddefocusblurredimage.Undertheassumptionofuniformlateralmotionofthecameraduring niteexposuretime,boththepinholemodelandthecamerawitha niteapertureareconsidered.Itisshownthattheimageblurproducedbyuniformlinearmotionofthecameraisinverselyproportionaltothedistanceoftheobject.Furthermore,ifthespeedoftherelativemotionisknown,thedepthoftheobjectcanbeacquiredbyidentifyingtheblurparameters.Animageblurmodelisformulatedbasedongeometricoptics.Theblurextentisestimatedbyintensitypro leanalysisandfocusmeasurementofthedeblurredimages.Theproposedmethodisveri edexperimentallyusingdifferenttypesoftestpatternsinanindoorenvironment.©2006SocietyofPhoto-OpticalInstrumentation

Engineers. DOI:10.1117/1.2403851

Subjectterms:motinblur;defocusblur;depthrecovery.

Paper050643RRreceivedAug.9,2005;revisedmanuscriptreceivedMay27,2006;acceptedforpublicationJun.5,2006;publishedonlineDec.11,2006.

1Introduction

Oneoftheessentialproblemsincomputervisionistore-coverthedistanceinformationofanobjectfromcapturedimages.Itsapplicationareasrangefromindustrialinspec-tionandreverseengineeringtoautonomousrobotnaviga-tionandcomputergraphics.Typically,thevisualcuesob-servedintherecordedimagesareusedfordepthperceptionofthescene.Forexample,the3-Dinformationcanbeen-codedinthetextureorshadinginformationoftheobject,imagedisparityfrommultipleviewpoints,depthof eldoftheoptics,etc.Thisworkaimstoaddresstheproblemofdepthrecoveryusingthevisualcuesprovidedbybothcam-eramotionandopticaldefocus.Speci cally,monlyusedtechniquesfordepthrecoveryincludeshapefromstereoormotion,shapefromshading,shapefromsilhouettes,andphotometricstereo.1Thesemethodsrequireeithermultipleimagescapturedfromdifferentviewpoints,ordifferentilluminationconditionsappliedforthesingleviewpointimageacquisition.Althoughitispos-sibletoachieveexcellent3-Dreconstructionresultsfrommultipleviewpoints,thecomputationalcostisconsiderablyexpensive.Inadditiontogeneraldepthrecoveryalgo-rithms,whichrelyonthechangesoftheenvironmentortheimagingposition,therearealsosomeothertechniquesthatutilizetheactiveadjustmentsoftheinternalcameraparam-eters.Someoftheproposedmethodsincludedepthfromzoominganddepthfromfocus/defocus.Thedepthinforma-tionisextractedbycomparingseveralimagesrecordedbyasinglecamerawithdifferentcameraparametersettings.Amotorizedzoomlensisusuallyrequiredtochangethezoomorfocuspositionsintheseapproaches.

Depthfromdefocusblurisaclassicapproachtorecover

thedistanceofanobjectandsimultaneouslyrestoreafo-cusedimageusingafewout-of-focusimages.2–4Theim-agesareusuallyobtainedfromasingleviewpoint,andthecamerahastoremainstaticduringimageacquisition.Thedepthoftheobjectisthencomputedusingtheamountofdefocusblurassociatedwiththerecordedimage.Recently,defocusimagecuehasalsobeencombinedwithstereotechniquesfordensedepthmaprecovery.Deschênes,Ziou,andFuchs5proposedauni edapproachforacooperativeandsimultaneousestimationofdefocusblurandspatialshiftsfromastereoimagepair.Basedongeneralizedmo-ment,expansion,theyhadformulatedasystemofequationsfordepthcomputation.Rajagopalan,Chadhuri,andMudenagudi6modeledthedepthandthefocusedimageindividuallyasMarkovrandom elds.Theyusedtwode-focusedstereoimagepairstoobtainadensedepthmapaswellasafocusedimageofthescene.

Differentfromopticaldefocus,motionblurisaresultof niteacquisitiontimeofpracticalcamerasandtherelativemotionbetweenthecameraandthescene.Forarapidscenechange,itisnotnegligibleevenwithapinholecam-eramodel.Traditionally,imagedegradationcausedbymo-tionbluristreatedasanundesirableartifactandusuallyhastoberemovedbeforefurtherprocessing.7Recently,motionblurhasbeenusedinvariousapplicationssuchassurveil-lancesystems,8computeranimation,9increasingthespatialresolutionofstillimagesfromvideosequences,10ormea-suringthespeedofamovingvehicle.11Althoughtheresto-rationtechniquesforimagedegradationcausedbymotionblurhavebeeninvestigatedforthepastfewdecades,12–14fairlylittleworkhasbeendonefordepthrecoveryusingmotionblurcues.Someofthepreviousresearchdealingwithcamera orobject motionandopticaldefocuscanbefound,forexample,inRefs.15–17.

ThemostrelevantresearchtothisworkisprobablytheworkgivenbyMylesanddaVitoriaLobo.15Theypre-

One of the essential problems in computer vision is to recover the distance information of an object from captured images. Its application areas range from industrial inspection and reverse engineering to autonomous robot navigation

LinandChang:Depthfrommotionanddefocusblur

blursimultaneouslyfromanimagepair.Amodelwaspro-posedandusedtoobtaintherelationshipbetweentheaf netransformationandthelevelofblur.Thederivedequationwasthensolvedusinganiterativeapproach.Althoughtheexperimentalresultswerepresentedwithseveralrealandcomplicatedimages,onlythechangesindefocusduetothemotionalongtheopticalaxiswereconsidered.Motionblurcausedbytherelativemotionbetweenthecameraandthesceneduringnonzerocameraexposuretimewasnotexplic-itlytakenintoaccount.Furthermore,theirformulationbasedonanaf nemodelrequirestwoimagesforbothde-focusblurandaf nemotionrecovery.Inthecurrentre-search,wearemoreinterestedindepthrecoveryfromasingleblurredimage.Theimageblurismodeledasopticaldefocusand/ortheimagedegradationcausedbytheobjectmotionduringnonzeroimageacquisitiontime.Thus,somepracticalissuesrelatedtotheproblem,suchastheimageformationandmotionblur,arenotcompletelyaddressedRef.15.

InFox’searlywork,16theconceptofrangefromablur-acutestereopairwasintroduced.Aspecialhardwarewasdesignedtocaptureboththeblurredandunblurredimagesfordepthmeasurements.Itwasbasicallyaconventionalstereomethodincorporatedwithtranslationmotionblurin-formation.Moreover,theauthormainlydescribedthethe-oreticalaspectoftheproposedideawithoutexperimentalresults.Recentworkrelatedto3-DreconstructionfromblurimagecueswasgivenbyFavaro,Burger,andSoatto.17Theyproposedavariationalapproachtorecoverthedepthmapandradianceofasceneusingadefocusedandmotion-blurredimagesequence.Theblurinformationwasusedtominimizethediscrepancybetweenthemeasureddefocusedimagesandtheoutputsynthesizedbythediffusionprocess.Althoughanoff-the-shelfcameracanbeused,multipleim-agesarestillrequiredforanisotropicdiffusion.Further-more,theirapproachisonlyvalidundertheassumptionofLambertiansurfacewithuniformillumination.

Incontrasttothepreviouswork,weuseonlyasinglemotionanddefocusblurredimagefordepthrecovery.Depthcalculationisaccomplishedbyidentifyingtheblurextentcausedbylateralcameramotionanddistance-varyingdefocus.Inthisresearch,animageblurmodelforuniformlinearmotionandopticaldefocusisformulatedbasedongeometricoptics.Theblurextentoftheimageisthenestimatedandusedfordepthmeasurement.

Theworkorganizedasfollows.Section2introducesthetheoryofimageformationwithdefocusandmotionblur.InSec.3,wepresentthemethodsforblurparameterestima-tionandcamerafocuscalibration.Section4providestheimplementationdetailsandexperimentalresults.Severaltypesofrealimageswithdifferentexperimentalsetupsareusedtovalidatetheproposedtechnique.Finally,Sec.5concludestheworkandpointstopossibledirectionsoffu-tureresearch.

2TheoreticalFormulation

Asimplecameramodelconsistingofathinlensandanimageplaneisusedtoderivesomefundamentalcharacter-isticsoffocusingbasedongeometricoptics.TheexistingalgorithmsfordeterminingobjectdepthfromimagefocusordefocususuallyformulatetheopticalblurduetotheFig.1Cameramodelfordefocusblur.

thiswork,boththeimageblurcausedbythecameramotionandopticaldefocusaretakenintoaccount.We rstmodelthemotionbluronly,i.e.,theobjectisinfocusifthereisnorelativemotionbetweenthecameraandthescene,andthenconsiderthedefocusbluraswell.

2.1GeometricCameraModel

Inthissectionwebrie ydescribethecameramodelforthegeometricimageformationprocess.InterestedreaderscouldrefertoRefs.21and22formoredetailedinforma-tiononthistopic.AsillustratedinFig.1,acamerasystemconsistingofasingleconvexlenswithfocallengthfisconsidered.TherelationshipbetweenthepositionofapointPinthesceneandthecorrespondingfocusedpositionP intheimageisgivenbythewellknownlensformula111+=,pqf

1

wherepisthedistanceoftheobjectfromthelensononeside,andqisthedistanceoftheimageplanefromthelensontheotherside.

IfweconsideranobjectpointQwithadistancezfromthelens withoutlossofgenerality,weassumezisgreaterthanp ,thenEq. 1 canberewrittenas111+=,zz f

2

wherez isthedistanceofthevirtualfocuspositionfromthelens.Furthermore,thecorrespondingimagepointofQismodeledasablurcirclecenteredatCaccordingtogeo-metricoptics.FromEq. 2 andtherelationdq z =,Dz

thediameterdoftheblurcircleisgivenbyd=Dq

3

111

,fzq

4

whereDisthediameterofthelensandqisthedistancefromthelenstotheimageplane.23Sincethedistanceqis

One of the essential problems in computer vision is to recover the distance information of an object from captured images. Its application areas range from industrial inspection and reverse engineering to autonomous robot navigation

d=

Dpf11

.

p fpz

5

Thatis,thesizeoftheblurcircleforanyscenepointlo-catedatadistancezfromthelenscanbecalculatedbyEq. 5 .

SimilarderivationsofEq. 5 canbefound,forexample,inRefs.20and24.Itisclearthattheblurcirclediameterddependsonlyonthedepthzif xedcamerasettingsofD,f,andparegiven.Inthiswork,wemakeseveralfurtherobservationsontheimagingmodel.First,Eq. 5 canberewrittenasd=c wherec=

Df

.p f

7

s =s cos +sin tan .

10

cp,z

6

Fig.2Cameramodelforpuremotionblur.

FromEq. 6 ,thesizeoftheblurcircleislinearlyrelatedtotheinversedistanceoftheobject.Moreover,theblurcirclediameterd→0asthedistancez→p,andconverselyd→casz→ .Inthelattercase,theconstantcgivenbyEq. 7 representsthemaximumdiameteroftheblurcirclewhentheobjectapproachesin nity.

IfwerewriteEq. 6 asafunctionofd,thenthedepthzoftheobjectisgivenbyz=cp

.c d

8

SubstituteEq. 9 withEqs. 10 and 1 ,andthedistancepisgivenby

s

p=1+ cos +sin tan f,

x

11

whichisindependentofthedistanceqbetweenthelensandtheimageplane.

Sincetheangle isusuallysmallifonlythecentralpartoftheimageisconsidered,especiallyforp f,Eq. 11 canbeapproximatedbyp=1+

s

cos f.x

Inthepreviousequation,thefocusingrangepcorrespond-ingtoa xedlenspositionatqfromtheimageplanecanbeobtainedfromcamerafocuscalibration.Thisisalsoanim-portantobservation,sincetheconstantccanbeeithercom-puteddirectlyfromEq. 7 withaperturediameterofthecameraormeasuredfromthesizeoftheblurcircleastheobjectapproachingin nity.Consequently,thedistancezcanbeacquiredfromEq. 8 byobservingtheblurcirclesize.

2.2ModelforMotionBlur

Differentfromopticaldefocus,motionblurisaresultoftherelativemotionbetweenthecameraandthesceneduringtheimagingprocess.AsshowninFig.2,supposeanobjectmovesadistancesfromPtoQ,andtheanglebetweenthemotiondirectionoftheobjectandtheimageplaneofthecamerais .Ifwefurtherassumethatthe3-DpointQisinthesamedepthof eldasthepointP,sothatthereisnodefocusblurassociatedwiththeimagepointQ whentheobjectPisinfocus.ThenthedistancebetweenQ andthelensisgivenbyq,andwehaves x=,pq

9

12

SupposethedistancebetweenQ andtheimagecenteris ,thentan =

11

=

qfpf

ifp f.

Foracharge-coupleddevice CCD sensorsizeof11mm

diagonalandacamerafocallengthof9mm,tan islessthan7 10 2mmifQ appearsinsidethecentral20%oftheimage. Thus,thedepthcanbecomputedbytherelativedisplacementandmovingdirectionoftheobjectmotion.Furthermore,iftheangle isnotsigni cant,i.e.,therela-tivemotionisapproximatelyparalleltotheimageplane,thenEq. 12 canbewrittenasp=1+

sf.x

13

bysimilartriangles.Let betheanglebetweenQQ andtheopticalaxisofthelens,thenthedisplacementscanbeItshouldbenotedthatifasimplepinholecameramodelisconsidered,thedepthpisgivenbysf/x.Thus,thereexistsadifferenceofthefocallengthfwhencomparedtothecameramodelwitha niteaperture.

Now,supposetheobjectundergoesuniformlinearmo-tionduringtheimagingprocesswithcameraexposuretimeofTseconds,i.e.,theobjectmovesfromPtoQatthe

One of the essential problems in computer vision is to recover the distance information of an object from captured images. Its application areas range from industrial inspection and reverse engineering to autonomous robot navigation

s x=,zq

18

wherexisthedifferencebetweenP andQ ,thecentersoftheblurcirclesassociatedwithPandQ,respectively.SimilartothederivationsofEqs. 11 and 12 ,thedis-tancezoftheobjectisgivenbyz=

spf cos +sin tan vTpf cos +sin tan

=,

x p f x p f

19

or

Fig.3Cameramodelforbothmotionanddefocusblur.

z=

vT

cos f,p=1+x

spfcos vTpfcos

=

x p f x p f

if 0deg, 20

14

andp=1+

vT

f,x

15

wherepisthefocusingrangecorrespondingtothedistanceqbetweenthelensandtheimageplane.Thatis,thedis-tancezcanbeobtainedprovidedthatalloftheparametersf,v,p,T,x,and areknown.Forthecasethattheobject’smotiondirectionisparalleltotheimageplane,Eq. 20 canbefurthersimpli edtoz=

spfvTpf

=,

x p f x p f

21

respectively,wherexcorrespondstotheblurextentduetotherelativemotion.Sincethefocallengthandexposuretimearegivenbythecamerasettings,thedistanceoftheobjectcanbeobtainedfromEqs. 14 or 15 ifthemovingspeedvoftheobjectisknownandtheextentofmotionblurxcanbeidenti ed.Moreover,foranytwoobjectswiththesamemotiondirectionandmovingspeedwithrespecttothecamera,theirrelativedepthcanbecomputedbyp2x1vTcos +x2

=,p1x2vTcos +x1

16

providedthattheobjectsarelocatedwithinthesamedepthof eld i.e.,out-of-focusblurcanbeignored .

Inmostpracticalsituations,thedisplacementoftheob-jectss=vTismuchlargerthantheblurextentsx1andx2intheimage.Thus,Eq. 16 canbefurtherreducedtop2x1

=.p1x2

17

bysetting aszero.DifferentfromEqs. 11 – 15 ,whichdealwiththecasewithoutdefocusblur,theparameterpinEqs. 19 – 21 representsthefocusingrangewithrespecttoa xedlenspositionwiththedistanceqfromtheimageplane.Itcanbeobtainedfromfocuscalibrationfor xedparametersettingsofthecamera.25However,theparameterxrepresentingthedisplacementbetweenthecentersofde-focusblurcirclesappearsmoredif culttoidentify.

FromEq. 20 ,itisclearthattherelativedepthoftwoobjectsatdifferentdistancesisgivenbyz2x1

=,z1x2

22

Thatis,exceptfortheextentsofmotionblur,neitherthecameraparametersnorthemovingspeedarerequiredforthecalculationofrelativedepth.Althoughtheimagingmodelassumestheobjectsareinmotion asshowninFig.2 ,therelativemotioncanbeachievedbymovingthecam-eralaterally.

2.3ModelforBothMotionandDefocusBlur

AsshowninFig.3,ifweconsideranobjectPwithadistancezfromthecamera assumingzislargerthanp ,thenthecorrespondingimagewillbeablurcirclecenteredatP andthecirclesizeisgivenbyEq. 4 .SupposetheobjectmovesfromPtoQwithadisplacementsandanwithouttheassumptionsx1,x2 srequiredforEq. 17 .FromboththeEqs. 17 and 22 ,therelativedepthoftwoobjectsisinverselyproportionaltothemotionblurextents,providedthattherelativemotiondirectionisthesame.Ifweconsideraspecialcasewheretheobjectisveryfaraway,orthefocusingrangeissetasin nity i.e.,p→ ,thenEq. 19 canbesimpli edasz=

vTf

cos +sin tan ,x

23

whichcorrespondstoapinholecameramodel.Inthiscase,theparameterxisnotonlythedistancebetweenthecentersoftheblurcircles,butalsotheblurextentwithoutthepres-enceofdefocusblur.

Ifboththedefocusandmotionblurareconsidered,theblurextentcausedbymovinganobjectfromPtoQisgivenbythedisplacementofthecentersofblurcirclesplus

One of the essential problems in computer vision is to recover the distance information of an object from captured images. Its application areas range from industrial inspection and reverse engineering to autonomous robot navigation

LinandChang:Depthfrommotionanddefocusblur

diametersoftheblurcirclesassociatedwithP andQ ,respectively.Thentheblurextentcausedbybothmotionandopticaldefocusisx+

d1+d2

,2

24

2.4PointSpreadFunction

Theobservedimageg x,y iscommonlymodeledastheoutputofa2-Dlinearspace-invariantsystem,whichischaracterizedbyitspointspreadfunction PSF ,h x,y .Moreprecisely,thedegradedimageg x,y canbeformu-latedasg x,y =

wherexisthedisplacementoftheblurcircles,andd1andd2aregivenbyEq. 4 .IfthedepthchangebetweenPandQisnotsigni cant,Eq. 24 canbeapproximatedbyx+d,wheredistheaverageofd1andd2.Inpractice,theGaussianblurmodelisextensivelyused4,6,18,23,26andtheblurparameterisgivenby =kd,wherekisacameraconstant.

Supposetheobservedblurextentisx intheimage,i.e.,x =x+kd,thenthedepthzcanbederivedifEq. 20 canbewrittenasafunctionofx .SubstitutingEq. 4 withEqs. 1 and 20 gives

fxDpf11

=D,d=

p fpzp fscos

h x ,y f , d d , 30

whereh x,y isalinearshift-invariantPSF,andf x,y is

theidealimage.Ifweconsideralosslesscamerasystem,then

h x,y dxdy=1, 31

25

andthePSFofimagedegradationcausedbyout-of-focusblurcanbewrittenasapillboxfunction4d222

,ifx+y

4,h x,y = d2

0,otherwise

wheredispositiveifz pandnegativeifz p.Forthe

formercase,Eq. 25 canberewrittenasx=

scos kDf

x ,

scos kDp f

32

26

sincex =x+kd.Finally,thedepthzoftheobjectisgivenbyz=

scos kD pf vTcos kD pf

=,

x p f kDfx p f kDf

27

accordingtogeometricoptics,wheredisthediameterof

theblurcirclegivenbyEq. 5 .

Foranimagewithanidealstepedge,thecorrespondingblurimageisgivenbythe1-Dconvolutiong x =f x *l x ,

wherel x isthelinespreadfunction4d if x d 4x,

2.l x = d2

0,otherwise

33

fromEq. 20 .Similarly,wehavez=

scos +kD pf vTcos +kD pf

=,

x p f +kDfx p f +kDf

34

ifzissmallerthanp.

Therearesomeobservationsfromthepreviousequa-tions.First,ifapinholecameramodelisconsidered i.e.,D→0 ,thenx=x byEq. 26 .Consequently,Eq. 27 isreducedtoEq. 20 .Second,ifthedisplacementoftheobjectismuchlargerthanthesizeofthecameraaperture,i.e.,s D,thenxcanbeapproximatedbykDf

,x x

p f

28

However,duetoaberrations,diffraction,andnonidealitiesofthelenses,aGaussianPSFiscommonlyusedinsteadofEq. 32 ,andthelinespreadfunctionisgivenbyx2

l x =

2 exp 2 2,

1

35

where isthespreadparameter.

Itisshownthat isproportionaltotheblurcircledi-ameterd,i.e.,

whichisindependentoftheangle .Furthermore,itcanbereducedtokDf

,x x p

29

=kdfork 0, 36

andcanbedeterminedbyanappropriatecalibrationprocedure.27Thus,Eq. 8 canberewrittenasz=c p

,c

37

iff p.FromEq. 29 ,itisclearthatdefocusblurisnegligibleiftheaperturesizeandfocallengtharerelativelysmallcomparedtothedisplacementoftheobjectandthe

One of the essential problems in computer vision is to recover the distance information of an object from captured images. Its application areas range from industrial inspection and reverse engineering to autonomous robot navigation

Fig.4Intensitypro lesofdefocusededgesaccordingtopillboxandGaussian

models.

kDf

,c =

p f

38

basedontheGaussianlinespreadfunctionmodel.Thein-tensitypro lesofdefocusededgesaccordingtothepillboxfunctionandGaussianmodelareshowninFig.4.Foredgeimages,ablurcircleisreducedtoahorizontalblurlength.ThelengthcanbeestimatedfrombluranalysisonstepedgesandusedinEq. 37 fordepthrecovery.

3FocusCalibrationandBlurExtentEstimation3.1DefocusBlurandFocusCalibration

TocalculatethedepthofanobjectusingEqs. 8 or 37 foragivensetofcameraparameters,thecorrespondingfocusingrangepandtheblurextentsc and havetobeidenti ed.AssuggestedbyEq. 1 ,thefocuspositionqcan

bederivedforanarbitrarydistancepoftheobject.Thus,adistancepcanbeassignedandusedto ndthecorrespond-inglenspositionatq,whichwillbe xedforblurextentestimationanddepthrecovery.Theproblemofdepthrecov-eryisthendividedintotwosubproblems—focuscalibra-tionforthefocusingrangepandblurestimationfortheparameterscandd.

Forthefocuscalibration,aplanarobjectwithastepedgeisplacedinfrontofthecameraata xeddistance.Asequenceofimagesiscapturedwithdifferentlensposi-tions.Thebestfocusedimageandthecorrespondinglenspositionareselectedbytheimagewiththelargestaveragegradientchangeintheedgedirection.Forthe xedlensposition,theobjectlocationisthenslightlyadjustedto ndthebestfocusingrangep.Inthisfocuscalibrationstage,theobjectisusuallyplacedclosetothecamerabecause

the

One of the essential problems in computer vision is to recover the distance information of an object from captured images. Its application areas range from industrial inspection and reverse engineering to autonomous robot navigation

correspondingdepthof eldissmallerforgeneralopticalsystems.Thedepthcorrespondingtothemaximumblurextentisalsoshorterinthiscase,whichgivesmoreaccu-rateblurparameterestimationforthein nitedistance.Givena xedfocusposition,theblurparameterscorc correspondingtothein nitedistancecanbeobtainedbyeitherdirectcomputationusingEq. 7 withknownfocallengthandaperturediameter,orblurextentestimationbyplacingacalibrationpatternatin nity.IftheGaussianmodelisconsidered,theparameterc hastobederivedusingthesecondmethod.Forthe rstmethod,assuggestedbyEq. 7 ,theprecisionofccanbeincreasedbysettingalargeraperturediameterDandasmallerdistancep.3.2MotionBlurEstimation

ToestimatetheblurextentduetotherelativemotionandopticaldefocususedinEqs. 19 , 23 ,and 27 fordepthmeasurements,atwo-stepalgorithmisproposed.Withoutlossofgenerality,weassumethedirectionofmotionblurisparalleltotheimagescanlines.Forthegeneralcasethattherelativemotionisnotparalleltotheimagescanlines,themotiondirectioncanbeidenti ed rstandthenusedtorectifytheimage.14

Itiswellknownthattheresponseofasharpedgetoanedgedetectorisathincurve,whereastheresponseofabluredgetothesameedgedetectorspreadsawiderregion.Thissuggeststhattheresultofedgedetectioncanbeusedasinitialestimationofthemotionblurextent.Intheimple-mentation,averticalSobelmaskisusedtoidentifythehorizontalblurextentforeachverticaledgealongtheim-agescanlines.Duetonoiseandtextureoftheobject,thederivedblurextentsareusuallynotallidentical.Thus,themodeoftheblurextentsisusedtoderiveanoverallhori-zontalblurextentoftheimage,andthenusedasaninitialestimateforthenextstep.

Atthesecondstep,theinitialblurextentisusedasaparametertocreateasequenceofdeblurredimagesbyaWiener lter.Amodi edLaplacianfocusmeasure25isthenusedtoselectthebestdeblurredimageandthecorrespond-ingblurextent.Ifopticaldefocusismodeledaswell,thedefocusblurextentdgivenbyEq. 25 isalsoconsideredforimagedeblurring.Fromx =x+dandEq. 25 ,thede-focusblurextentcanbewrittenasd=

Fig.6Theedgeimageusedinthe

experiments.

placedonastagemountedonalinearrailtoobtainedthemotionblurredimages.ThemovementofthestageisdrivenbyaPC-controlledsteppingmotor.

4.1OpticalDefocus

Toobtainthemaximumblurextent correspondingtotheobjectatin nity ,focuscalibrationfortwodifferentfocus-ingrangesisperformedwithtwodifferentaperturesizes.Inthe rstexperiment,thefocusingrangeissetas265mmfromthecamera.Theblurextentsestimatedfromdifferentobjectdistanceswithfocallengthsof36mmareshowninFig.8.Solidanddottedlinesrepresenttheresultswithaperturediametersof7.2and15mm,respectively.Theplotshowsthat,astheobjectdistancegoestoin nity,theblurextentsapproach79and163pixelsforthesetwoaperturesizes.Theresultsalsoverifythatthemaximumblurispro-portionaltotheaperturediameterasderivedinEqs. 7 and 38 .DirectcomputationsoftheblurextentcusingEq. 7 aregivenby166.5and346.8pixelsfortheaperturediam-etersof7.2and15mm,respectively.Thus,theconstantkgivenbyEq. 36 isabout0.47forbothcases.

Inthesecondexperiment,thefocusingrangeissetas1000mmfromthecamera.Thefocallengthissetas36mm.Figure9showstheblurextentsobtainedfrom

dif-

fx sD .p fss kD

39

Sincethefocusmeasurehighlydependsonlocalintensityvariation,thedefocusblurextentdcanbeusedtodeter-minethenumberofdeblurredimagestobecreated.How-ever,motionblurdominatestheblurextentinmostcases,andtypicallynineimagesarecreatedformotiondeblurringintheimplementation.

4ImplementationandExperimentalResults

Theproposeddepthrecoverymethodhasbeentestedinthelaboratoryenvironment.Aschematicdiagramfortheex-perimentalsetupisshowninFig.5Twotypesofplanarpatterns,onewithastepedgeandtheotherwithafewcharacters asshowninFigs.6and7,respectively ,are

One of the essential problems in computer vision is to recover the distance information of an object from captured images. Its application areas range from industrial inspection and reverse engineering to autonomous robot navigation

LinandChang:Depthfrommotionanddefocusblur

Table1Depthrecoveryfromdefocusblur withthefocusingrangep=265mm .

Aperture15mm

Distance

mm 50060070080090010001100

Fig.8Blurextentsfordifferentobjectdistances focusat265mm ;solidanddottedlinescorrespondtoaperturediametersof7.2and15mm,

respectively.

12001300

Blur668699107113115126127129130133

Depth44655967877085990311791201127712881438

Error10.76.73.03.74.49.67.10.11.77.94.1

Aperture7.2mmBlur3439454854535659636464

Depth4615266126718528059291027126813761376

Error7.712.212.416.05.219.415.514.32.41.68.2

ferentobjectdistancesfortheaperturediametersof7.2mm solidlines and15mm dottedlines ,respectively.Theplotindicatesthatthemaximumblurextentsapproach24and50pixelsforthesetwocameraparametersettings.Di-rectioncomputationsgiveblurextentsof39.5and82.4pixels,whichyieldthecameraconstantkof0.61forbothcases.

Fordepthfromdefocusblur,thecameraiscalibratedsuchthatthefocusingrangeis265mmfromthecamera,asdescribedinSec.3.1.Atestpatternwithanidealstepedgeisusedtoobtaintheblurextentsfordifferentdistances.Theobjectisplacedatdistancesof500to1500mmfromthecameraforevery100mmapart.Twoexperimentsarecar-riedoutwiththeaperturediametersof7.2and15mm.ThedepthsarecomputedusingEq. 37 withthefocusingrangep=265mm,andthemaximumblurextentsc =79and163

14001500

pixelsfromfocuscalibration.Table1showstheblurex-tents inpixel ,recovereddepths inmillimeters ,andrela-tiveerrors inpercent intheexperiments.TheresultsarealsoillustratedinFig.10,whichindicatesthatthereisalinearrelationbetweentheactualandestimateddepthsinthedistanceof500to1500mm.

4.2MotionBlur

Fordepthfrommotionanddefocusblur,thecameraisinstalledsothatthemotiondirectionisperpendicularto

the

Fig.9Blurextentsfordifferentobjectdistances focusatFig.10Objectdistanceversusthecomputeddepth;solidanddot-

One of the essential problems in computer vision is to recover the distance information of an object from captured images. Its application areas range from industrial inspection and reverse engineering to autonomous robot navigation

LinandChang:Depthfrommotionanddefocusblur

Fig.11Motionblurredimage.

opticalaxis.Boththecameramotiondirectionsparallelandnonparalleltothetestpatternarecarriedoutintheexperiments.

4.2.1Parallelcameramotion

Inthiscase,thecameramotionisparalleltothetestobject.Thelenspositionofthecameraisadjustedsuchthatthefocusingrangeis900mm.Ablackandwhitepatternisplacedat500,750,1000,1250,and1500mmfromthecamerafordepthrecovery.Foursetsofexperimentswithtwodifferentcameramotionspeeds 115and230mm/s andtwodifferentaperturediameters 0.8and1.8mm areperformed.Thecameraexposuretimeis xedas1/5stoobservedifferentscalesofblurextentsfordifferentcameramotionspeeds.Therestofthecameraparametersaregivenasfollows:focallengthf=9mmandCCDpixelsizes=6.8 m.

Figure11showsanexampleofablurimagecapturedwiththecameramotionspeedof115mm/s,exposuretimeof1/5s,andaperturesizeof0.8mm.Thecorrespondingimageintensitypro leisillustratedinFig.12.Theblurextentsoftheimages,recovereddepths,andabsoluteerrorsrelativetothemeasureddistancesareshowninTables2and3.Thecolumnswithx,z1,z2,e1,ande2representtheblurextent inpixel ,recovereddistance inmillimeter ,andabsoluteerrors inpercent ofdifferentdistancescal-culatedusingEqs. 20 and 27 ,respectively.Oneinterest-ingobservationisthattheresultsgivenbyEq. 20 arevery

Fig.13Experimentalsetupfornonparallelcameramotion.

closetothosegivenbyEq. 27 inmostcases.Thissug-geststhatmotionblurmightdominatetheblurextentintheexperiments.

Anotherfoursetsofexperimentsarecarriedoutusingthe“character”image asshowninFig.7 .Onlytheimagescanlineswithverticaledgesareusedforblurextentesti-mation.Thecameraparametersettingsarethesameasthosedescribedinthepreviousexperiments.TheresultsofdifferentobjectdistancesaretabulatedinTables4and5.Fromtheexperimentalresults,nosigni cantdifferencescanbefoundbetweenthesetwotypesoftestimagepat-terns.Withthe xedcameraexposuretime,theblurextentisapproximatelyproportionaltotherelativemotionspeedasexpected.

4.2.2Nonparallelcameramotion

TheexperimentalsetupforthecameramotiondirectionnonparalleltothetestobjectisshowninFig.13.Theanglebetweentheobjectpatternandthelinearrailis15deg.FoursetsofexperimentswiththesameparametersettingsasdescribedinSec.4.2.1areperformedfordifferentimagepatterns.Thecenteroftheobjectis xedat500,750,1000,1250,and1500mmfordepthrecovery.Duetothenonpar-allelrelativemotionbetweentheobjectandthecamera,the

Table2Depthrecoveryfrommotionanddefocusblur withthe“edge”image .Thecameramotionspeedis115mm/s.

Aperture0.8mm

Distance

mm 50075010001250

x583932

z1527788963

z

2530790964

e1

e2

x

Aperture1.8mmz1491769

z2500773

e1

e2

5.36.1635.05.340

1.70.02.53.1

3.73.630103610423.64.2

27113911448.98.526116311767.05.9

One of the essential problems in computer vision is to recover the distance information of an object from captured images. Its application areas range from industrial inspection and reverse engineering to autonomous robot navigation

“edge”image .Thecameramotionspeedis230mm/s.

Aperture0.8mm

Distance mm 500750100012501500

x11777574640

d1527794

d2531796

e1

e2

x

Aperture1.8mmd1520811

d2524813

e1

e2

Distance mm 500750100012501500

x613931

“edge”image .Thecameramotionspeedis115mm/s.

Aperture0.8mmz1490761949

z2494763950

e1

e2

x

Aperture1.8mmz1492761

z2501766

e1

e2

5.46.31185.96.2

76584639

4.04.88.28.4

2.11.3601.51.739

1.50.21.52.1

107910837.98.3134613597.78.7154615683.14.6

106910736.97.3133013436.47.4157716015.16.7

5.15.028104610534.65.3

25118111875.65.025121112293.11.722137413888.47.521143814764.11.6

Table4Depthrecoveryfrommotionanddefocusblur withthe“character”image .Thecameramotionspeedis115mm/s.

Aperture0.8mm

Distance mm 500750100012501500

x5738

z1542815

z2545816

e1

e2

x

Aperture1.8mmz1540781

z2548785

e1

e2

Table7Depthrecoveryfromnonparallelcameramotion withthe“edge”image .Thecameramotionspeedis230mm/s.

Aperture0.8mm

Distance mm 500750100012501500

x11675574538

z1510797

z2515799

e1

e2

x

Aperture1.8mmz1519777

z2523779

e1

e2

8.39.1578.68.839

8.09.74.24.7

2.02.91156.36.5

76554538

3.74.63.63.9

30103610393.63.928106010686.06.823131513255.26.023131513405.27.219160216256.88.320157516255.08.3

104010444.04.4130613184.55.5156115854.15.7

108310888.38.8133413486.77.8154715713.24.7

Table5Depthrecoveryfrommotionanddefocusblur withthe“character”image .Thecameramotionspeedis230mm/s.

Aperture0.8mm

Distance mm 50075010001250x118755744z1519818

z2524820

e13.99.1

e2

x

Aperture1.8mmz1512809

z2517811

e1

e2

Table8Depthrecoveryfromnonparallelcameramotion withthe“character”image .Thecameramotionspeedis115mm/s.

Aperture0.8mm

Distance mm 50075010001250x5637

z1533803

z2537804

e1

e2

x

Aperture1.8mmz1573803

z2546806

e1

e2

4.71209.38.8

7556452.53.37.98.1

6.67.4557.07.237

7.59.27.07.5

108310888.3108910948.99.4133713507.08.029101510181.51.829103110373.13.724124412530.50.223129113153.35.2136713819.310.5

One of the essential problems in computer vision is to recover the distance information of an object from captured images. Its application areas range from industrial inspection and reverse engineering to autonomous robot navigation

“character”image .Thecameramotionspeedis230mm/s.

Aperture0.8mm

Distance mm 500750100012501500

x10974564537

z1545805

z2549807

e1

e2

x

Aperture1.8mmz1547803

z2551805

e1

e2

Acknowledgment

ThesupportofthisworkinpartbytheNationalScienceCouncilofTaiwanundergrantNSC-93-2218-E-194-024isgratefullyacknowledged.

8.99.81097.47.6

74564436

9.410.27.15.47.5

7.35.88.7

106610716.67.1131113234.95.9160116276.78.5

1054105813441358

References

1.E.TruccoandA.Verri,IntroductoryTechniquesfor3-DComputerVision,PrenticeHall,EnglewoodCliffs,NJ 1998 .

2.M.SubbaraoandG.Surya,“Depthfromdefocus:Aspatialdomain

approach,”put.Vis.13,271–294 Dec.1994 .

3.A.N.RajagopalanandS.Chaudhuri,“Simultaneousdepthrecoveryandimagerestorationfromdefocusedimages,”IEEEComputerVi-sionPatt.Recog.1,348–353 1999 .

4.P.FavaroandS.Soatto,“Ageometricapproachtoshapefromdefo-cus,”IEEETrans.PatternAnal.Mach.Intell.27 3 ,406–417 2005 .5.F.Deschênes,D.Ziou,andP.Fuchs,“Anuni edapproachfora

simultaneousandcooperativeestimationofdefocusblurandspatialshifts,”put.22 1 ,35–57 2004 .

6.A.N.Rajagopalan,S.Chaudhuri,andU.Mudenagudi,“Depthesti-mationandimagerestorationusingdefocusedstereopairs,”IEEETrans.PatternAnal.Mach.Intell.26 11 ,1521–1525 2004 .

7.M.BanhamandA.Katsaggelos,“Digitalimagerestoration,”IEEE

SignalProcess.Mag.14,24–41 Mar.1997 .

8.S.Kang,J.Min,andJ.Paik,“Segmentation-basedspatiallyadaptive

motionblurremovalanditsapplicationtosurveillancesystems,”Intl.Conf.ImageProcess.,pp.245–248 2001 .

9.G.BrostowandI.Essa,“Image-basedmotionblurforstopmotionanimation,”SIGGRAPH01Conf.Proc.,pp.561–566 Aug.2001 .10.M.Ben-EzraandS.Nayar,“Motion-basedmotiondeblurring,”IEEE

Trans.PatternAnal.Mach.Intell.26,689–698 June2004 .

11.H.Y.Lin,“Vehiclespeeddetectionandidenti cationfromasingle

motionblurredimage,”IEEEWACV/MOTION,pp.461–467 2005 .12.D.Slepian,“Restorationofphotographsblurredbyimagemotion,”

BellSyst.Tech.J.46,2353–2362 Dec.1967 .

13.M.Sondhi,“Imagerestoration:Theremovalofspatiallyinvariant

degradations,”Proc.IEEE60,842–853 July1972 .

14.Y.Yitzhaky,I.Mor,ntzman,andN.Kopeika,“Directmethod

forrestorationofmotionblurredimages,”J.Opt.Soc.Am.A15,1512–1519 June1998 .

15.Z.MylesandN.daVitoriaLobo,“Recoveringaf nemotionand

defocusblursimultaneously,”IEEETrans.PatternAnal.Mach.In-tell.20,652–658 June1998 .

16.J.Fox,“Rangefromtranslationalmotionblurring,”IEEEComputer

VisionPatt.Recog.1,360–365 1988 .

17.P.Favaro,M.Burger,andS.Soatto,“Sceneandmotionreconstruc-tionfromdefocusedandmotion-blurredimagesviaanisotropicdif-fusion,”inLect.NotesComput.Sci.3021,257–269 2004 .

i,C.W.Fu,andS.Chang,“Ageneralizeddepthestimation

algorithmwithasingleimage,”IEEETrans.PatternAnal.Mach.Intell.14 4 ,405–411 1992 .

19.N.Asada,H.Fujiwara,andT.Matsuyama,“Edgeanddepthfrom

focus,”put.Vis.26,153–163 Feb.1998 .

wrence,“Aninvestigationofmethodsfordetermin-ingdepthfromfocus,”IEEETrans.PatternAnal.Mach.Intell.15 2 ,97–108 1993 .

21.M.BornandE.Wolf,PrinciplesofOptics,6thed.,PergamonPress,

Oxford,England 1980 .

22.E.Hecht,Optics,4thed.,PearsonEducation,ISBN0805385665

2001 .

23.M.SubbaraoandN.Gurumoorthy,“Depthrecoveryfromblurred

edges,”IEEEComputerVisionPatt.Recog.1,498–503 1988 .24.M.Subbarao,T.Choi,andA.Nikzad,“Focusingtechniques,”Opt.

Eng.32 11 ,2824–2836 Nov.1993 .

25.S.NayarandY.Nakagawa,“Shapefromfocus,”IEEETrans.Pattern

Anal.Mach.Intell.16,824–831 Aug.1994 .

26.A.Pentland,“Anewsensefordepthof eld,”IEEETrans.Pattern

1651168010.112.0

scenedistanceperpendiculartotheimageplaneisnotaconstantfordifferentimageacquisitionpositions.Thus,themotionblurredimageiscapturedwhenthecamerareachesthedepthmeasurementposition i.e.,theplacerightinfrontoftheobject’scenter .

Tables6and7showtheexperimentalresultsusingthe“edge”image,withcameramotionspeedsof115and230mm/s,respectively.Theresultsofthe“character”im-agearetabulatedinTables8and9withtwodifferentcam-eramotionspeeds.Equations 20 and 27 areusedfordepthcomputationwiththeangle =15deg.

FromtheexperimentalresultsshowninTables2–9,mostoftherelativeerrorsarelessthan10%.Theresultssuggestthatthereisalinearrelationbetweenthetruedis-tanceandtherecovereddepth.Basedontheproposedcam-eramodel,theparametersgivenbythedepthrecoveryfor-mulasaretheerrorsources.Inadditiontothepossibleerrorcausedbytheincorrectnessofthecameraparameters,theaccuracyofthemotionspeed controlledbythesteppingmotor andblurextentestimationwillalsoaffectthecor-rectnessofthedepthmeasurement.

5ConclusionandFutureWork

Findingthedistanceofanobjectinasceneisanessentialprobleminmanyapplications.Severalapproachesbasedondifferentvisualcuesobservedfromimageshavebeenpro-posedinthepastfewdecades.Inthiswork,wepresentanovelapproachfordepthrecoveryfromasinglemotionanddefocusblurredimage.Differentfrompreviousdepthfromdefocusapproaches,whichusedatleasttwoblurredimages,ourmethodrequiresonlyasingledefocusedimagetorecoverthedepthofascene.Furthermore,thecalibrationstageforourdepthrecoveryalgorithmismuchsimplercomparedtothegeneraldepthfromfocusapproaches.Itisshownthat,inmostcases,defocusblurcanbeignoredinthepresenceofmotionblur.Infuturework,robustblurparameterestimationformorecomplexsceneswillbede-veloped.Depthmeasurementofoutdoorsceneswillbecar-riedoutbycapturingtheblurredimagesfrommovingob-

One of the essential problems in computer vision is to recover the distance information of an object from captured images. Its application areas range from industrial inspection and reverse engineering to autonomous robot navigation

LinandChang:Depthfrommotionanddefocusblur

Huei-YungLinreceivedhisBSdegreeinappliedmathematicsfromNationalChiaoTungUniversity,Taiwan,andMSandPhDdegreesinelectricalengineeringfromStateUniversityofNewYorkatStonyBrook.In2002hejoinedtheDepartmentofElectricalEngineering,NationalChungChengUniver-sity,Taiwan,asanassistantprofessor.Hiscurrentresearchinterestsincludecomputervision,digitalimageprocessing,andpatternrecognition.HeisamemberoftheIEEE

Chia-HongChangreceivedhisBSdegreeinindustrialeducationfromNationalTaiwanNormalUniversity,Taiwan,andMSdegreeinelectricalengineeringfromNationalChungChengUniversity,Taiwan.Hehasbeenengagingintheresearchareasofcomputervisionandimageprocessing.

and

SPIE.

本文来源：https://www.bwwdw.com/article/3h91.html