An MPEG-Processor-based Robot Vision System for Real-Time Detection of Moving Objects by a

This paper describes a PC-based vision system that can be used to detect moving objects from a mobile robot. An image processing board equipped with an MPEG motion estimation processor calculates a sparse but robust optic flow in real-time. An algorithm to

AnMPEG-Processor-basedRobotVisionSystemforReal-TimeDetectionof

MovingObjectsbyaMovingObserver

NorbertO.St¨of erandZoltanSchnepf

LaboratoryforProcessControlandReal-TimeSystems

TechnischeUniversit¨atM¨unchenD-80333M¨unchen,Germany

stof er@lpr.ei.tum.de,www.lpr.ei.tum.de/stof er/

Abstract

ThispaperdescribesaPC-basedvisionsystemthatcanbeusedtodetectmovingobjectsfromamobilerobot.AnimageprocessingboardequippedwithanMPEGmotiones-timationprocessorcalculatesasparsebutrobustoptic owinreal-time.Analgorithmtoevaluatethiskindofoptic owhasbeenrealizedinsoftware.Itdeterminesrelevantmotionparametersandasimplesceneinterpretationintermsofmovingobjectregions.Theimageprocessingboardandthealgorithmarepresentedinsomedetail;theperformanceofthesystemisdemonstratedbyexperiments.

1.

Introduction

Usingourexperimen-talrobotMARVIN(MobileAutonomousRobotwithVIsion-basedNavigation,seeFig.1),wearecurrentlyworkingontheautono-mousexplorationofof ce-typeenvironments.Primar-ilyvisionisusedtobuild3Dworldmodels[2].Im-portantcuestotheinterpre-tationofanobservedscenecanbeobtainedbytheeval-uationofmotion.Movingobjects(typicallypeopleintheabove-mentionedenvi-ronment)havetobedistin-Figure1.MARVINguishedfromstaticpartsof

thesceneontheonehandandhavetobetakenintoaccountfordynamicmotionplanningontheotherhand.Thisdetec-tionofmovingobjectshastotakeplaceduringthemotion

oftherobotitself.

Three-dimensionalmotionisprojectedontoatwo-dimensionalvelocity eld(ordisplacement eldinthecaseofisochronoussampling)ontheimageplaneofthecamera.Thisvector eldiscommonlytermedoptic ow(ormorepreciselyimage ow).Ifthedisplacementvectorsareregardedtobeequivalenttothevelocityvectors(anac-ceptableassumptionforrealisticframeratesandvelocities[1])andthefocallengthofthecameraisnormalizedto1(amathematicalconveniencethatdoesnotrestrictgenerality),theoptic ow

canbecalculatedaccordingtothefollowing:

This paper describes a PC-based vision system that can be used to detect moving objects from a mobile robot. An image processing board equipped with an MPEG motion estimation processor calculates a sparse but robust optic flow in real-time. An algorithm to

2Real-timeoptic owsensor

TheMPEGcompressionstandardsusethespatio-temporalredundancyinanimagesequenceforband-widthreduction.Ifpossible,onlydisplacementvectorsaretransmittedinsteadofthecompletepixelinformation.Al-thoughtheMPEGstandardsdonotregulatehowthesevec-torshavetobecomputed,thestate-of-the-arttechniqueiscorrelation.Ascorrelationiscomputationallyexpensive,specializedprocessors,socalledMEPs(MotionEstimationProcessors)haveevolvedfromthisarea.Areferenceblock(RB,16x16pel)fromimageiscomparedwithasearchwindow(SW,32x32pel)intheimage.Forallpossi-bleoffsets

,acorrelation-likevaluecalledSAD(SumofAbsoluteDifferences)iscalculated;themini-mumdesignatesthebestmatch,anditspositionde nesthedisplacementvector:

(2)

(3)

Ifconsecutiveimagesarecompared(i.e.

),theresultingvector eldcanberegardedasoptic ow.

TheideatouseoneofthoseextremelyoptimizedMEPsforthegenerationofoptic owisnotnew.InparticularInoueetal.describetheintegrationofaMEPintotheirim-ageprocessingtransputernetwork[5].Resultingfromtheirwork,acommercialversionisavailable,andmeanwhileisusedinvariousresearchprojects[3,4].

Aproblemthatseveralresearchersreportisthattheoptic owgeneratedbysuchacorrelationprocessorcanbecomeverynoisy.ThishappenswhentheimagestructureoftheRBsorSWsisambiguousorcompletelymissing.Thenthedetectionofasigni cantminimumaccordingtoEqn.3fails.

Fig.2aillustratestheproblem.The owwasgeneratedbyalinearforwardmovementofthecamera,soallvectorsshouldintersectinasinglepoint,theFOE(FocusOfEx-pansion).Duetolocallackofstructure,mostvectorspointtocompletelydifferentdirections.Unfortunately,thiseffectdominatesinmostof ce-typeenvironments.Furtherevalu-ationofsucha ow eldisvirtuallyimpossible.

Tosolvethisproblem,weaugmentedaMEPwithexter-nalcircuitrythatcalculatesanadditionalcon dencevalueforeachvector[7].Inthesimplestcase,thiscon dencevaluecanbetestedagainsta xedthresholdtosiftoutthenoisy owvectors.Adrawbackisthesparsenessofthere-maining eld(seeFig.2b).

OurprototypesystemconsistsofanISAimageprocess-ingboardcontainingtheMEP,andaLINUXhostPC(seeFig.

3).

Figure2.Optic ow,generatedbyaMEP.a)complete,b)sifted.

Figure3.Optic owsensor:Systemstruc-ture.

Threeframememoriesallowthecomparisonoftwoim-agesandsimultaneousacquisitionofthenextimageintothethirdmemory.DependingontheutilizationoftheMEPinternalpipeline,upto525vectorscanbecalculatedperframe(PAL,25Hz).Forapplicationsliketrackingmulti-

This paper describes a PC-based vision system that can be used to detect moving objects from a mobile robot. An image processing board equipped with an MPEG motion estimation processor calculates a sparse but robust optic flow in real-time. An algorithm to

pleobjectsorbigdisplacements(astypicallyinducedbycamerarotation)a xedblock-rasterisnotadequate.Toachievethehighestpossible exibility,thecoordinatesofRBsandSWscanberandomlysetbythesoftwarerunningonthePCforeachsinglematchingoperation.Alistof

positions

isreadviaDMA,thelistofresultsiswrittenbackbyDMAagain.ThissavesmostoftheCPU’sprocessingpowerfortheapplicationsoftwarethatevaluatesthegenerated ow eldsasdescribedinthenextsection.Foramoredetaileddescriptionofthehardwaresee[7].

3Detectionofmovingobjectsbyoptic owsegmentation

Manypapersintheliteratureofoptic owaddresstheproblemofobjectsegmentationandmotionparameterre-construction.Algorithmstocalculateall veparameters(theabsolutevalueofthetranslationvectorgetslostdur-ingthe3Dto2Dprojection)wereproposedforexamplebyPrazdny[6]orWengetal.[8].Adivpresentsanelegantap-proachtosolvethesegmentationproblemforall veparam-eters:Objectsareconsideredtoconsistofplanarsurfaces,sovectorscanbeclusteredinan8D(5Dforthemotionand3Dforthesurfaceparameters)Houghspace[1].

Applyingthesetechniquestooptic owscalculatedbyoursensorsystemproducesnosatisfyingresults,though.ResponsibleforthefailureoftheseverygeneralapproachesisthenumericalinstabilityoftheclosedformsolutiontoEqn.1alongwiththestrongquantizationerrorsofthecal-culatedvectors.

Thereasonforthisfailurecanalsobegraphicallyde-ducedbythesimilarityof eldsgeneratedbymerelateralandmererotationalmotion.Thoughallvectorsarepar-allelinthetranslationalcase,andalignedwithhyperbolas(accordingtoEqn.1)intherotationalcase,thedifferencehasthesameorderofmagnitudeasthequantizationeffects.Fig.4demonstratestheproblemforrealisticcameraparam-etersandconstant

depth.

Figure4.Optic ow:a)horizontaltranslation

b)verticalrotation.

Therefore,ageneralsolutiontothecompletemotionre-covery(i.e.segmentationofmovingobjectsanddetermina-tionofall5parametersforeachobject)seemsimpossible

inourcontext.

Morepragmaticapproachesthatcluster owvectorsalongtheir2DpropertiesasforexampleproposedbyYa-mamotoetal.[9],producedbetterresults,butaredif culttoadaptforamovingobserverandcannotbeusedforob-jectsmovingalongtheopticalaxis.

3.1Determinationofego-motion

Becauseoftheaboveproblems,weintroducesomesim-pli cationstothegeneralapproachesthatareinspiredbytherequirementsofourapplication.Becausethecameraismountedonamobilerobotwithnon-holonomickinemat-ics,onlyonetranslationalandonerotationalparameterre-main.Withoutfurtherrestrictinggenerality,buttosimplifytheequations,itisassumedthatthecameraismountedhori-zontally.Incameracoordinates,thereis

andthegeneralequationoftheoptic ow(Eqn.1)canbereducedtothefollowing:

and)remain,

Eqn.4canbesolvedforeachvectorasfollows:

(5)

isameasurementofthedistanceof

thecorresponding3Dpointandthushasanindividualvalueforeachvector(itisthereciprocalvalueoftheTimeToCollision,TTC).

Thesecondresult

shouldcorrespondtotherotationalvelocityofthecameraandthereforebeidenticalforeachvector.Thus,therotationofthecameracanbeestimatedbyasimplemeanvaluecalculation:

(7)

ExperimentsonMARVINinarealof ce-typeenviron-menthavedemonstratedthefeasibilityoftheproposedes-timationapproachfor.Whenemployingthemedianin-steadofthesimplemeanvalue,therobustnessofthecal-culatedvalueagainstremainingfaultyvectorscanbein-creasedfurther.Fig.5showsacomparisonoftheestimatedandtheonecalculatedbytheodometryoftherobot.Sincetheresultingvectorlengthsigni cantlyexceedsthemaximalvectorlengthoftheMEP,evenformoderateturningratesoftherobot(

This paper describes a PC-based vision system that can be used to detect moving objects from a mobile robot. An image processing board equipped with an MPEG motion estimation processor calculates a sparse but robust optic flow in real-time. An algorithm to

0.4

0.4

0.350.35

0.30.3

0.250.25

0.20.2

0.150.15

0.10.1

0.050.05

050100150200250

050100150200

parisonofa)theestimatedandb)thecorrespondingodometricvalue

Takingintoaccountonlythevectorsof,theestimate

forthecamerarotationcanagainbecalculatedaccordingtoEqn.7.Fig.6demonstratestheperformanceoftheback-grounddetectionstatisticallyusingtypicalimagesequencescontainingonetoseveralmovingobjects.Inthisexperi-ment,thecameraisnotmoving,i.e.theestimatedshouldbe.Eachdotcorrespondstoone ow eld.Theab-scissadepictsthepercentageof owvectorsbelongingtothebackground,theordinatetheestimated(herein

Simplyusingthelast

tocalculateabiasvector

transformsthelimitationoftherotationalvelocitytoa

limitationoftherotationalacceleration.Thus,agoodesti-mateisessentialfortheoveralloperationofthevisionsystem.

3.2Segmentationofmovingobjects

Inthecaseofoneormoremovingobjectsintheobservedscenethedeterminationoftherotationhastobemodi ed.

ofvectorsbelongingtoamovingobjectnolongerThe

ofthecamera.Segmentationcanbecoincidewiththe

performedbyclusteringvectorsaccordingtotheir.Asimpleclusteralgorithmhasproventobesuf cientinthe

,experiments.Vectorsaresortedbyincreasingvaluesof

so

Thentheyarecombinedtoformasetofclusters

aslongastheysatisfythecriterion.Therefore

.

clustersbuildthe rsthypothesesfortheseg-These

mentationoftheoptic ow.Unfortunately,noteachclustercorrespondstoonemovingobject,becausetheassumptionsthatledtoEqn.4areonlytrueforarestrictedcameramo-tion,i.e.forthevectorsbelongingtothebackground.Sointhisstep,onlyaclassi cationbackgroundversusnotback-groundcanbemadeforeachcluster.Goodresultscanbeachievedbydeclaringtheclusterasbackgroundthatcontainsthesetofvectorswiththelargestspatialvariance.Thisapproachisinspiredbytheobservationthattheback-groundistheonly”object”notcorrespondingtoacompactimageregion.

Since,asintheliveexperimentsonlythe rst eldofeachframeisused,thevarianceindirectionismuchmoremeaningfulthanindirectionandcanbeusedexclu-sively:

This paper describes a PC-based vision system that can be used to detect moving objects from a mobile robot. An image processing board equipped with an MPEG motion estimation processor calculates a sparse but robust optic flow in real-time. An algorithm to

Figure7.Segmentationresults(PAL, rst eldonly)

ofthoseboundingboxesinalist,hypothesesofobjectre-gionscanalsobemaintainedwhenthedetectionfailsforashorttime,i.e.whenamovingobjecttemporarilystops.Fig.7showssamplesfromatypicalimagesequenceusedintheexperiments.Thecamera(i.e.therobot)ismov-ingalongaslightleftturnwhileapersoncrossesthescene.Vectorsnotbelongingtothebackgroundarerobustlyde-tectedandclustered.Theboundingboxescan,ofcourse,onlyencompassthosepartsofthemovingobjectswhereoptic owvectorshavebeencalculatedandhavepassedthesiftingprocess.

4Conclusionandfurtherwork

Wehavepresentedalowcostbutef cientimagepro-cessingsystemthatisabletocalculateasparseoptic owinreal-time(upto525vectorsperframe).Further,aprac-ticalalgorithmtodetectmovingobjectsbysegmentationofthe ow eldwasproposedandsomeexperimentalresultswerepresented.Thesystemiscurrentlyusedinclosedloopexperimentsonanautonomous,vision-guidedrobot.

Furtherworkwillbeconcernedwithimprovementstotherobustnessofthesystem.ApplyingKalman lteringtotheestimationofthecamerarotationcanfurtherreducethesensitivitytonoiseandallowapredictionofevenwhenthebackgrounddetectiontemporarilyfails.Forthevectorsbelongingtothebackground,theadditionalparameter

本文来源：https://www.bwwdw.com/article/uko4.html