Augmented Reality Scouting for Interactive 3D Reconstruction

AugmentedRealityScoutingforInteractive3DReconstruction

BernhardReitinger 1

1

ChristopherZach2DieterSchmalstieg1

InstituteforComputerGraphicsandVision,GrazUniversityofTechnology,Austria

2VRVisResearchCenter,Austria

ABSTRACT

Thispaperpresentsa rstprototypeofaninteractive3Drecon-structionsystemformodelingurbanscenes.AnAugmentedRe-alityscoutisapersonwhoisequippedwithanultra-mobilePC,anattachedUSBcameraandaGPSreceiver.Thescoutisexplor-ingtheurbanenvironmentanddeliversasequenceof2Dimages.TheseimagesareannotatedwithaccordingGPSdataandusediter-ativelyasinputfora3Dreconstructionenginewhichgeneratesthe3Dmodelson-the- y.Thisturnsmodelingintoaninteractiveandcollaborativetask.

Keywords:scouting,interactive3dreconstruction,urbanplanning1

INTRODUCTIONANDRELATEDWORK

Generating3Dmodelsofoutdoorscenesinurbanenvironmentsisoftenademandingtask,butnecessaryforapplicationssuchasmobileAugmentedReality(AR)[3,12],interactivevisualiza-tion[2,9],ormodel-basedtracking[13].Creatingthesemodelsisusuallydoneinanof ineprocess,usingconventional3Dmodelingtools,andinvolvestedioushoursofmanualdatapreparation.

Incontrast,manyinterestingapplicationsdemandthatmodelsmustbecreatedon-lineandon-site.Forexample,urbanplannersliketospontaneouslyexperimentwithvariationsoftheirarchitec-turaldesignswheninspectingaplannedconstructionsite.Thismeansthatthe3Dmodelgenerationmustbeperformedinterac-tivelytogiveimmediatefeedback.Ingeneral,mostapplicationsthatrequiredigitalreconstructionofarchitecturecanbene tfromimmediatefeedbackthatallowstoverifythereconstructionpro-cess.Providingthisinteractivityistheaimoftheworkpresentedinthispaper.

However,traditionalreconstructiontechniquesareaimedto-wardsahigh-accuracyoff-lineworkstyle,wheredataacquisitionanddataprocessingarestrictlyseparated.Theobjectiveofsuchsystemsistoobtainthebestscalabilityoftheoverallprocessbyfullautomationoftheacquisitionandreconstructionphase.Forexample,Akbarzadehetal.[1]usegeo-registeredvideosequencescapturedbyamulti-camerasetupmountedonavehicle.Thisdataisthenpost-processedinaseparateoff-linestageand nallygener-ates3Dmodelsofthecapturedenvironment.

Anotherapproachonlyusesaerialimagesforreconstructionur-banscenes[8].Area-basedsegmentationisusedtoclusterhomo-geneousphotometricpropertiesandcalculateadensemaptoob-tainthereconstruction.Thismethodcanbeusedtoreconstructlarge-scalearchitecturalscenes.However,highqualityaerialim-agesmustbeavailable.AnotherapproachispresentedbyWangetal.wherethetextureoffacadesisreconstructedbasedonanumberofphotographs[16].

AdifferentapproachcalledPhotoTourismwaspresentedbySnavelyetal.[14].Inthisproject,similarimagesofthesame

e-mail:

reitinger@tugraz.at

IEEE Virtual Reality Conference 2007

March 10 - 14, Charlotte, North Carolina, USA1-4244-0906-3/07/$20.00 ©2007 IEEE

buildingaretakenfromanexistingdatabaseandprocessedinor-dertogenerateasparse3Dpointcloud.Thecamerapositionsandorientationsarereconstructedforeachimageandimage-basedren-deringisprovided.Sincethissystemaimsatlotsofsimilarimages,theprocessingtimefordozensofimagesisbeyondonehour.Oncethe3Dreconstructionis nished,theresultcanbeobservedinaninteractiveviewer.

Alargebodyofworkonreconstructionalgorithmscanbefoundintheroboticscommunitybutwillnotbediscussedhere.Roboticsaswellasallthereconstructionworksmentionedaboveaimatau-tomated3Dreconstruction;noneofthembringsthehumanintothereconstructionloop.WeproposetoemployahumanARscoutwhoisabletospontaneouslyexploreandreconstructenvironmentswhicharenotyetknown.Theresultingmodelscanimmediatelybeinspectedandre nedbythescoutorusedbyabroaderremoteaudiencethroughawirelessconnection.Thistransformstheusu-allyoff-linemodelingtaskintoaninteractivetaskwhereagroupofpeopleandthescoutgeneratemodelson-the- y.

2SYSTEMOVERVIEW

Ourproposedinteractivereconstructionsystemconsistsoftwomainsub-systems,thescoutandthereconstructionserver(seeFig-ure1):

Figure1:Anoverviewoftheinteractive3Dreconstructionsystem.TheARscoutstorescurrentpositionandimagedatainthedatabase.Thereconstructionenginegetsanoti cationandcalculatesthe3Dmodelwhichisagainstoredinthedatabase.Finally,theresultcanbevisualizedforabiggeraudienceonaprojectionscreen.

TheARscoutacquiresgeo-referencedimagedatawithahand-heldARdeviceanddeliversittoaremotereconstructionserver.Theserverisresponsibleforprocessingtheindividualimagesinto

219

a3Dmodel(texturedpointcloud).Reconstructionisaverycom-putationallyintensivetaskandcannotbecarriedoutwithsuf cientperformancebyamobilecomputer.Theserveralsostorestheac-quiredandreconstructeddataandmakesitinstantlyavailabletoremoteusers.Thereconstructedmodelisreturnedtothescoutforimmediate3D-registereddisplayandinspectiononthehandheldARdevice.Ifde cienciesaredetected,thescoutcanusetheARdevicetoacquiremoredataorpruneerroneousreconstructions.TheARscoutisequippedwithanultra-mobilePC,anattachedGPSreceiver,andaUSBcamera.Whileexploringtheenviron-ment,thescouttakesseveralimagesforinstanceofatargetbuild-ing.Theseimagesareautomaticallyannotatedbycurrentposition-ingdata(takenfromtheGPSreceiver).Theenricheddataisthentransmittedtoacustomdatabasestore[15].Thisdatabaseisde-signedformulti-userdataexchangebasedonthedocumentobjectmodel.

Wheneveranewimageisstoredinthedatabase,thereconstruc-tionenginegetsanoti cationandtriggersthereconstructionpro-cess(detailedinSection3).Theenginerequiresatleastthreedif-ferentviewsinordertogenerateaninitial3Dmodel.Eachfurtherimageisaddedinaniterativewayandupdatesthemodelaccord-inglywithinseconds.Again,thedatabaseserverisusedforstoringthe3Dmodel.

TheARscoutequipmentmustbelight-weight,connectedtoanetworkandequippedwithsensors.Oursetupconsistsofahand-heldultra-mobilePC(SamsungQ1)withatouchscreenandafront-mountedcamera.TheARuserinterface1wasdevelopedusingtheStudierstubesoftwareframework.Figure2(a)showsthefrontandthebacksideofthe

handheld.

Bluetooth

GPS receiver

USB 1.3 Mpixcamera

(a)FrontviewoftheSamsungQ1showsthelivevideocapturedbytheUSBcamera.Atiponthedisplaytriggersthecaptureroutine.TheUSBcameraand

theGPSreceiveraremountedonthebackside.

GPStransmissionconfidence

notification

GPSnumber ofcoordinatessatellites

(b)Thestatusbarofthecaptureapplicationcontainsimportantfeedbackfortheusersuchascurrentpositionorcon denceofthesignal.

Figure2:TheARscoutsetupisusedforcapturingannotatedimagedatainanurbanenvironment.

Astatusbar(showninFigure2(b))displaysfeedbackonloca-tion,qualityoftheGPSsignal,numberofsatellites,andatrans-1

220

missionnoti cation.Theuserpointsthedeviceatthetargetloca-tionlikeadigitalcamera,andtriggerstheimagecapturewhicharetransmittedtothedatabasetogetherwithcorrespondingGPSinfor-mationusingaWLANor3Gconnection.TheGPSreceiverwithWAAS(wideareaaugmentationsystem)typicallyhasaprecisionof2-5meters.Threeormoreimagesofalocationinthedatabasetriggerthereconstructionprocedure.3

RECONSTRUCTIONENGINE

Thereconstructionengineactsasablackboxwhichtakes2Dim-agesanddelivers3Dmodels.Themainideaisthatasequenceof2Dimages(containingasuf cientoverlapinimagecontents)isusedto ndcorrespondencesbetweenthem.Thesecorrespon-dencescanthenbeusedtoestimatethecamerapositionswherethe2Dimageweretaken.Themathematicalframeworktogenerate3Dgeometryfrommultipleimagesispresentedin[6].Oncetheinitialmodelisknown,consecutiveimagescanberelatedtoeachother,andatextured3Dpointcloudcanbecomputedbyadensematch-ingapproach.Inthefollowing,abriefoverviewofeachindividualtaskisgiven.Theengine’spipelineisshowninFigure3.3.1CameraCalibration

Thereconstructionengineonlyworksforcalibratedcameras.Forthisreasontheintrinsiccameraparameters(focallengthf,andtheprincipalpoint(px,py))aredeterminedusingatargetcalibrationprocedureinadvance(e.g.[7]).Incaseof xedlenses,thecalibra-tionprocedureisneededtobeperformedonlyonce.Inadditiontothecameraintrinsicparametersfand(px,py)theutilizedlensmayhaveasigni cantdistortioneffectontheimage,putingtheundistortedimagefromtheoriginaloneisbasedonalook-uptableobtainedfromthedistortionparam-eters.Oncethecameraiscalibrated,theinformationispassedontotheengine.Smalldeviationsofthecamerafromthedeterminedcalibrationresultscanbeaddressedlaterbythebundle-adjustmentprocedure(Section3.3).3.2FeatureExtraction

Sincetheinputimagescontaintoomuchredundantinformationforactualthereconstruction,themostrelevantinformationrequiredfor ndingcorrespondencesmustbeextractedbyusingfeaturepoints.Featureextractionselectsimagepointsorregionswhichgivesignif-icantstructuralinformationtobeidenti edinotherimagesshow-ingthesameobjectsofinterest.WeuseHarriscornersasfeaturepoints[5]whicharewellsuitedforsparsecorrespondencesearcheswhichisthecaseforurbanscenes.

3.3CorrespondenceandPoseEstimation

Inordertorelateasetofimagesgeometricallyitisnecessaryto ndcorrespondences.Forthetaskofcalculatingtherelativeorientationbetweenimagesitissuitabletoextractfeatureswithgoodpointlocalizationasprovidedbythefeatureextractionstep(seeabove).Therelativeorientationbetweentwoviewstakenfromcalibratedcamerascanbecalculatedfrom vepointcorrespondences.HenceaRANSAC-basedapproachisusedforrobustinitialestimationoftherelativeposebetweenthe rsttwoviews.Weutilizeanef cientprocedureforrelativeposeestimation[11]inordertotestmanysamplesquickly.Theresultofthisprocedureistherelativeorien-tationbetweenthesetwoviews,butwithunknownoverallscale.Therelativeposetranslatesintoknownepipolargeometry,whichrepresentstherelationshipofpixelsinoneviewwithimagesofthecorrespondingcameraraysinthesecondview.Withtheknowledgeoftherelativeposesbetweentwoviewsandcorrespondingpointfeaturesvisibleinatleast3images,theorientationsofallviewsinthesequencecanbeupgradedtoacommoncoordinatesystem

Figure3:Thereconstructionpipelineconsistsoffourmaincomponents:featureextraction,correspondencesearch,cameraposeestimation,anddensematching.Eachcapturedimageispassedthroughthispipelineinordertogenerateorenhancethe3Dmodel.

byusinganabsoluteposeprocedure[4],againcombinedwithaRANSACapproachtoincreasetherobustness.Theposeofeveryadditionalincomingimageiscalculatedonthebasisof2Dto3Dpointcorrespondences,whichisstrongerthanusingtheepipolarrelationshipbetweentwoviewsalone.

Purelyimagebasedreconstructionsarelocatedinalocalcoordi-natesystem,whichisupgradedtoaworld-referencesystemusingthemeasuredGPSlocationsofthecamerapositions.2Thetransfor-mationfromthelocaltotheglobalsystemisasimilaritytransforminthecaseofcalibratedcameras.

Thecameraposesandthesparsereconstructionconsistingof3Dpointstriangulatedfrompointcorrespondencesarere nedus-ingasimplebutef cientimplementationofsparsebundleadjust-ment[10].Ourimplementationallowsthere nementofthecameraintrinsicparametersandtheintegrationofGPSdatawithestimateduncertaintiesaswell.Theoutputofthisstepareoptimizedcameraorientationsandintrinsicparametersinthe rstplace.Addition-ally,sparse3Dpointscorrespondingtotheimagefeaturesvisibleinseveralviewsarere ned,too.3.4DenseDepthEstimation

Withtheknowledgeofthecameraparametersandtherelativeposesbetweenthesourceviewsdensecorrespondencesforallpixelsofaparticularkeyviewcanbeestimated.Sincetherelativeposebe-tweentheincorporatedviewsisalreadyknown,thisprocedureisbasicallyaone-dimensionalsearchalongthedepthraysforeverypixel.Triangulationofthesecorrespondencesresultsinadense3Dmodel,whichre ectsthetruesurfacegeometryofthecapturedob-jectinidealsettings.

WeutilizeaGPU-acceleratedplane-sweepapproachtogeneratethedepthmapforeachsourceview[17,18].Planesweeptech-niquesincomputervisionaresimpleandelegantapproachestoim-agebasedreconstructionwithmultipleviews,sincearecti cationprocedureasneededinmanytraditionalcomputationalstereometh-odsisnotrequired.The3Dspaceisiterativelytraversedbyparallelplanes,whichareusuallyalignedwithaparticularkeyview(Fig-ure4).Theplaneatacertaindepthfromthekeyviewinduceshomographiesforallotherviews,thusthereferenceimagescanbemappedontothisplaneeasily.

Iftheplaneatacertaindepthpassesexactlythroughthesur-faceoftheobjecttobereconstructed,thecolorvaluesfromthekeyimageandfromthemappedreferencesimagesshouldcoincideatappropriatepositions(assumingconstantbrightnessconditions).

theGPSantennaandtheprojectioncenterofthecameraare

very

close,weignoretheresultingoffsetbetweenthem.

2Since

Figure4:Planesweepingprinciple.Fordifferentdepthsthehomog-raphybetweenthereferenceplaneandthereferenceviewisvarying.Consequently,theprojectedimageofthereferenceviewchangeswiththedepthaccordingtotheepipolargeometry.

Hence,itisreasonabletoassignthebestmatchingdepthvalue(ac-cordingtosomeimagecorrelationmeasure)tothepixelsofthekeyview.Bysweepingtheplanethroughthe3Dspace(byvary-ingtheplanesdepthwrt.thekeyview)a3Dvolumecanbe lledwithimagecorrelationvaluessimilartothedisparityspaceimage(DSI)intraditionalstereo.Thereforethedensedepthmapcanbeextractedusingglobaloptimizationmethods,ifdepthcontinuityoranyotherconstraintonthedepthmapisrequired.Weemployasimplewinner-takes-allstrategytoassignthe naldepthvaluesforperformancereasons.3.5Output

Adepthmapisgeneratedasdescribedaboveforeverytripletofad-jacentviews,andthesingledepthimagesneedtobefusedintoonecommonmodel.Currently,weemployaverysimpletechnique:thedepthmapsareconvertedintocoloredpointclouds(usingthereferenceviewfortexturing),andthesepointsetsareconcatenatedtoobtainthecombinedmodel.Thisapproachallowsaneasyin-crementalupdateofthedisplayedmodelaftergenerationofanewdepthmap.Futureworkwilladdressthecreationof3Dsurfacemeshesfromthedepthmaps(e.g.[18]),whichrequiresmorecom-plexmethodstoassignatexturetotheresultingmodel.4

RESULTS

The rstprototypewastestedwithmultiplebuildingsatourcam-pus.Additionally,weusedittoreconstructanancientbrickwall

221

showninFigure5.Onlysomesmallcluttercanbeobservedinthebottomoftheresultingmodel.

Thereconstructiontimedependsontheimageresolutionandthenumberofextractedfeatures.Fortheaboveexample,eachpassofthepipelinetakeslessthanoneminuteforuploading,featureextraction, ndingcorrespondences,densematching,andupdatingthe3D

model.

Figure5:Thesescreenshotsshowatestdatasetofanoldbrickwall(with6inputimages).Theimageinthemiddleshowsthereconstruc-tionofthecamerapositionsincludingsparsepoints.Theimageonthebuttonshowsascreenshotofthe nal3Dmodel.

5CONCLUSION

ARscoutingallowson-linegenerationofarbitrary3Dmodelsinurbanenvironments.The rstprototypedeliverspromisingresultsandworkswellwithahandheldultra-mobilePC.Theresulting3Dmodelsarecurrentlyrepresentedbyatextured3Dpointcloud.DuetotheGPSinformation,thereconstructedmodelsareavailableinaglobalcoordinatesystemandcanberegisteredwithavailable3Dgeographicinformationsystems.

Inthenearfutureweplantoreplacethepointcloudmodelswithtruesurfacemeshesgeneratedbyarobustandincrementaldepthmapintegrationtechnique.Wealsointendtotestphysicallydistributedcollaborative3Dmodelingwithmultiplescoutsexplor-ingtheenvironmentsimultaneouslyandreconstructinglargerareas.Wealsointendtoperformadetailedquantitativeanalysisoftheob-tainedmodelsintermsoftheirreconstructionaccuracycomparedagainstconventionaloff-linereconstructiontechniques.

222

ACKNOWLEDGEMENTS

ThisresearchisjointworkbetweenGrazUniversityofTechnology(sponsoredpartiallybytheEuropeanUnionundercontractFP6-2004-IST-4-27571andtheAustrianScienceFundFWFundercon-tractY193)andVRVisResearchCenter(fundedbytheAustriangovernmentresearchprogramK-plus).REFERENCES

[1]A.Akbarzadeh,J.-M.Frahm,P.Mordohai,andetal.Towardsurban

3dreconstructionfromvideo.InInternationalSymposiumon3DDataProcessing,VisualizationandTransmission(3DPVT),2006.

[2]H.Benko,E.Ishak,andS.Feiner.Collaborativemixedrealityvisual-izationofanarchaeologicalexcavation.InProc.oftheInternationalSymposiumonMixedandAugmentedReality(ISMAR2004),2004.[3]S.Feiner,B.MacIntyre,T.H¨ollerer,,andT.Webster.Atouringma-chine:Prototyping3dmobileaugmentedrealitysystemsforexplor-ingtheurbanenvironment.InProc.oftheISWC’97(FirstIEEEInt.

Symp.onWearableComputers),pages208–217,1997.[4]R.M.Haralick,C.Lee,K.Ottenberg,andM.N¨olle.Analysisand

solutionsofthethreepointperspectiveposeestimationproblem.InIEEEComputerSocietyConferenceonComputerVisionandPatternRecognition(CVPR),pages592–598,1991.

[5]C.HarrisandM.Stephens.Acombinedcornerandedgedetector.

Proceedings4thAlveyVisualConference,pages189–192,1988.

[6]R.HartleyandA.Zisserman.MultipleViewGeometryinComputer

Vision.CambridgeUniversityPress,2000.[7]J.Heikkil¨a.Geometriccameracalibrationusingcircularcontrol

points.IEEETransactionsonPatternAnalysisandMachineIntel-ligence(PAMI),22(10):1066–1077,2000.[8]A.Huguet,R.Carceroni,andA.deA.Ara´ujo.Towardsautomatic

3dreconstructionofurbanscenesfromlow-altitudeaerialimages.InProc.ofthe12thInternationalConferenceonImageAnalysisandProcessing(ICIAP’03),2003.

[9]H.Ishii,J.Underkof er,D.Chak,B.Piper,E.Ben-Joseph,L.Yeung,

andZ.Kanji.Augmentedurbanplanningworkbench:Overlayingdrawings,physicalmodelsanddigitalsimulation.InProc.oftheInt.SymposiumonMixedandAugmentedReality(ISMAR),2002.

[10]M.LourakisandA.Argyros.Thedesignandimplementationof

agenericsparsebundleadjustmentsoftwarepackagebasedonthelevenberg-marquardtalgorithm.TechnicalReport340,InstituteofComputerScience-FORTH,Heraklion,Crete,Greece,August2004.[11]D.Nist´er.Anef cientsolutiontothe ve-pointrelativeposeprob-lem.IEEETransactionsonPatternAnalysisandMachineIntelligence

(PAMI),26(6):756–770,2004.

[12]W.PiekarskiandB.Thomas.Tinmith-metro:Newoutdoortechniques

forcreatingcitymodelswithanaugmentedrealitywearablecomputer.In5thInt’lSymposiumonWearableComputers,pages31–38,Zurich,2001.

[13]G.ReitmayrandT.Drummond.Goingout:Robustmodel-based

trackingforoutdooraugmentedreality.InProc.oftheInternationalSymposiumonMixedandAugmentedReality(ISMAR2006),2006.[14]N.Snavely,S.Seitz,andR.Szeliski.Phototourism:Exploringphoto

collectionsin3d.ACMTransactionsonGraphics(SIGGRAPHPro-ceedings),25(3):835–846,2006.

[15]D.WagnerandD.Schmalstieg.Muddlewareforprototypingmixed

realitymultiusergames.InProc.oftheIEEEVirtualReality2007,2007.

[16]X.Wang,S.Totaro,F.Taillandier,A.Hanson,andS.Teller.Recov-eringfacadetextureandmicrostructurefromreal-worldimages.InProc.2ndInternationalWorkshoponTextureAnalysisandSynthesis,pages145–149,2002.

[17]R.Yang,G.Welch,andG.Bishop.Real-timeconsensusbasedscene

reconstructionusingcommoditygraphicshardware.InProceedingsofPaci cGraphics,pages225–234,2002.

[18]C.Zach,M.Sormann,andK.Karner.High-performancemulti-view

reconstruction.InInternationalSymposiumon3DDataProcessing,VisualizationandTransmission(3DPVT),2006.

本文来源：https://www.bwwdw.com/article/n1pe.html