ARCam: an FPGA-based Augmented Reality Framework
Transcript of ARCam: an FPGA-based Augmented Reality Framework
![Page 1: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/1.jpg)
{jpsml, gfg, gds, jmxnt, ebmx, vt, jk} @cin.ufpe.br
ARCam: an FPGA-based Augmented Reality Framework
Petrópolis, May 2007
SVR 2007
JOÃO PAULO LIMAGermano Guimarães
Guilherme Silva
JOÃO MARCELO TEIXEIRAEmanoel Xavier
Veronica TeichriebJudith Kelner
![Page 2: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/2.jpg)
Motivation
• Many AR tracking techniques require image processing procedures
• Usually done by software– General purpose processing– Operating system overhead– Impact on frame rate and image
resolution– Increase on clock frequency and
power consumption
![Page 3: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/3.jpg)
Motivation
• Solution: embedded image processing– Dedicated hardware– Better performance– Real parallelism– Low power consumption– High resolution images
![Page 4: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/4.jpg)
Goals
• ARCam: framework for the development of embedded AR systems
• Library of common AR functions
• Development model based on components
![Page 5: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/5.jpg)
Related work
• It was not found any flexible embedded solution for AR applications
• Features of existing solutions– Rely on hybrid hardware-software approaches– Dedicated to specific applications
• ARCam contribution– Entirely hardware based– General component based framework for developing
embedded AR applications
![Page 6: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/6.jpg)
ARCam development environment
Digital image sensor
Altera Stratix II FPGA
![Page 7: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/7.jpg)
Framework architecture
ARCam architecture
![Page 8: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/8.jpg)
Framework architecture
ARToolKit Pipeline
Search formarkers
Find marker 3Dposition andorientation
Identifymarkers
Position andorient objects
Render 3D objectsin video frame
![Page 9: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/9.jpg)
Framework architecture
ARCam architecture
Search formarkers
Find marker 3Dposition andorientation
Identifymarkers
Position andorient objects
Render 3D objectsin video frame
![Page 10: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/10.jpg)
Framework architecture
ARCam architecture
![Page 11: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/11.jpg)
Implemented components
• Image binarization and gray scalingLabelingGeneric convolutionMean filterEdge detectionCentroid estimationQuad detectionHardwire
![Page 12: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/12.jpg)
Implemented components
Image binarization and gray scaling
• LabelingGeneric convolutionMean filterEdge detectionCentroid estimationQuad detectionHardwire
![Page 13: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/13.jpg)
Implemented components
Image binarization and gray scalingLabeling
• Generic convolutionMean filterEdge detectionCentroid estimationQuad detectionHardwire
N11 N12 N13
N21 Pin N23
N31 N32 N33
C11 C12 C13
C21 C22 C23
C31 C32 C33
333332323131
2323in222121
131312121111out
NCNCNCNCPCNCNCNCNCP
×+×+×+×+×+×+×+×+×=
![Page 14: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/14.jpg)
Implemented components
Image binarization and gray scalingLabelingGeneric convolution
• Mean filterEdge detectionCentroid estimationQuad detectionHardwire
![Page 15: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/15.jpg)
Implemented components
Image binarization and gray scalingLabelingGeneric convolutionMean filter
• Edge detectionCentroid estimationQuad detectionHardwire
![Page 16: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/16.jpg)
Implemented components
Image binarization and gray scalingLabelingGeneric convolutionMean filterEdge detection
• Centroid estimationQuad detectionHardwire
![Page 17: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/17.jpg)
Implemented components
Image binarization and gray scalingLabelingGeneric convolutionMean filterEdge detectionCentroid estimation
• Quad detectionHardwire
![Page 18: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/18.jpg)
Implemented components
Image binarization and gray scalingLabelingGeneric convolutionMean filterEdge detectionCentroid estimationQuad detection
• Hardwire
![Page 19: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/19.jpg)
Performance analysis
Ratio (100MHz)Process sw/hw
Binarization 18.89Gray Scale 16.103x3 Filter 30,428.57Mean Filter 1,751.15Edge Detection 1,951.06Labeling 3,623.64Centroid 5.26QuadDetection 27.63
![Page 20: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/20.jpg)
Results
• Two case studies– Pong
• Prototype AR application rapidly created• Do not worry about modularization
– Object recognition• Make use of the componentized design model
![Page 21: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/21.jpg)
Results :: Pong
![Page 22: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/22.jpg)
Results :: Object recognition
Centroid estimation Hardwire(x,y)
![Page 23: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/23.jpg)
Lessons learned
• Software to hardware translation
• Recursion to iteration
• New possible optimizations
![Page 24: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/24.jpg)
Conclusions
• An architecture was implemented to support the development of AR embedded solutions
• A pre-existent infrastructure makes the development of hardware based AR applications easier and faster
• Performance obtained from the hardware implementation was shown to be satisfactory
![Page 25: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/25.jpg)
Future work
• Performance analysis• Finish QuadDetector• Hardwire extension
– Z-buffer– Textures
• Creation of an authoring tool for hardware based AR applications
• More complex AR studies• Different AR approaches
– Markerless AR• External memory access
![Page 26: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/26.jpg)
Virtual Realityand MultimediaResearch Group
{jmxnt, ds2, gsm, lhcbc, vt, jk} @cin.ufpe.br
miva: Constructing a Wearable Platform Prototype
Petrópolis, May 2007
SVR 2007
João Marcelo TeixeiraDALITON SILVA
Guilherme Moura
Luiz Henrique CostaVeronica Teichrieb
Judith Kelner
![Page 27: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/27.jpg)
Introduction
• Virtual and augmented reality applications hosting
• Mobile and autonomous execution
![Page 28: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/28.jpg)
Introduction
• High performance hardware requirement– 3D applications– Visual quality/response time result in high cost!
• Minimum intervention on user’s mobility– Wearable computer
![Page 29: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/29.jpg)
miva Prototype
![Page 30: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/30.jpg)
miva Prototype
• Intel Pentium-M,1.4GHz• 512MB DDR (up to 1024MB)• 855GME Intel, 16-64MB• Intel 10/100 Mbit/s
![Page 31: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/31.jpg)
miva Prototype
• 4 USB V2.0 ports• Stereo analog output and SPDif audio
input, mic and SPDif• COM1/COM2 RS232• 5Volts DC 1.5Ah
![Page 32: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/32.jpg)
miva Prototype
36cm
3.4kilograms
31cm 4hours
![Page 33: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/33.jpg)
miva Prototype
• Software setup– Windows XP Embedded
• Reduced storage space required• High performance
![Page 34: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/34.jpg)
Software Architecture
![Page 35: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/35.jpg)
Software Architecture
• Application layer modules
![Page 36: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/36.jpg)
Software Architecture
• Service layer– Hardware abstraction API– Can be customized and incremented by user– Provides a high level abstraction for
communication and persistence services
![Page 37: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/37.jpg)
Software Architecture
• Support layer modules
![Page 38: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/38.jpg)
Potential Applications
mivaDesk
![Page 39: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/39.jpg)
Potential Applications
mivaDesk
![Page 40: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/40.jpg)
Potential Applications
mivaTherm
![Page 41: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/41.jpg)
Conclusions and Future Work
• RV/RA platform• Easily extensible• Developed without previous project
design
![Page 42: ARCam: an FPGA-based Augmented Reality Framework](https://reader030.fdocuments.in/reader030/viewer/2022012518/619263e687b4d72df0468970/html5/thumbnails/42.jpg)
Conclusions and Future Work
• miva platform physical evaluation– Size must be reduced– A more usable container will be developed– Use of more accurate pointer devices (data
glove and tracker)