© Copyright Khronos Group, 2009 - Page 2
Agenda
- …nothing works without them- Jon Peddie, JPR
The Khronos Handheld API Ecosystem- Neil Trevett, NVIDIA
EGL- Working Group report – Jon Leech, Khronos
OpenGL ES- Working Group report – Tom Olson, ARM- KTX: A light-weight file format for OpenGL and OpenGL ES textures – Mark Callow, HI Corp- Real-time, on-device OpenGL ES profiling using gDebugger – Yaki Tabaka, Graphic Remedy- OpenGL ES 2.0: 3D Graphics for In-Vehicle Infotainment – Petri Talala, Symbio
OpenVG- OpenVG Implementations and Applications – Hwanyong Lee, HUONE
Jon
Ped
die
Re
sea
rch
Agenda - a bunch of overviews
Status and Momentum of current APIs
• (It’s been a long, hard trail…)
Future developments and trends
• (You may be surprised, or maybe not)
Trends in the mobile graphics markets
• (Some interesting, others FYI)
Jon
Ped
die
Re
sea
rch
A little 30 year History of APIsAPI Devloper OS Release Date API Developer OS Release Date
Plot 10 Tektronix Tek 4010 & HG 1971
CORE ACM Various 1972
GKS ACM Various 1977
PHIGS ACM Various 1979
IRIS--GL SGI Various 1981
ADI Autodesk Various 1988 8514 IBM MS DOS 1988
PEX Various 1989
S3G S3 MS DOS 1991
Fusix PX/GL Du Pont Various 1991 GL Compaq (ACE)
MESA Various 1991 Glide 3Dfx MS DOS & others 1995
Open GL SGI Various 1992 QuickDraw 3D Apple Apple 1995
Direct3D Microsoft MS DOS 1995
PIKS ISO/IEC Various 1995 DirectX 1.0 Microsoft Windows 3.1 Sept. 1995
DirectX 2.0 Microsoft 95 and NT 4.0 Jun. 1996
DirectX 3.0 Microsoft NT 4.0 SP3 Sept. 1996
Fahrenheit Project DirectX 4.0 Microsoft n/a Never released
DirectX 5.0 Microsoft 98 Jul. 1997
GDI Microsoft 98 Jul. 1998
DirectX 6.0 Microsoft 98 SE and ME Aug. 1998
GLX SGI Various 1999 DirectX 7.0 Microsoft 2000 Sept. 1999
DirectX 8.0 Microsoft 2000 Nov. 2000
DirectX 8.1 Microsoft XP Nov. 2001
DirectX 9.0 Microsoft XP Dec. 2002
DirectX 9.0a Microsoft XP Mar. 2003
DirectX 9.0b Microsoft XP Aug. 2003
WGF (LDDM) Microsoft XP 2004
DirectX 9.0c Microsoft XP Aug. 2004
DirectX 10 Microsoft Vista Nov. 2006
PS3-RSX Sony/Nvidia PS3 2006
DirectX 10.1 Microsoft Vista SP1 Feb. 2008
DirectX 11 Microsoft Windows 7 Oct. 2009
Jon
Ped
die
Re
sea
rch
Future Trends
Application
Driver
Operating System
Hardware
API and Graphics Libraries
Jon
Ped
die
Re
sea
rch
The Mobile Market Overall
0
500
1,000
1,500
2,000
2,500
20092010
20112012
20132014
2015
598 691 784 895 1,031 1,197 1,335
1,0261,091
1,1141,141
1,129
1,1071,062
Mobile Devices WW annual shipments (M units)
All other phones
Mobile (non PC) devices
BILLIONS of thingies get suppliers excited…
Jon
Ped
die
Re
sea
rch
The Suppliers( There’s a lot… )
Main Suppliers Others
Broadcom *
Freescale *
Intel **
Marvell *
NEC *
Nvidia **
Qualcomm **
Renesas *
Samsung **
ST-Ericsson *
Telechips
TI **
Zoran
Anyka
MediaTek
Mtekvision *
Sigma Design
NetLogic *
Wondermedia/VIA *
IP Providers ARC/Virage/Synopsys
ARM **
DMP *
Imagination Technologies **
MIPS
Takumi *
Vivante ** KHRONOS Member
Jon
Ped
die
Re
sea
rch
The Sub or Adjacent Markets
0
200
400
600
800
1,000
1,200
1,400
20092010
20112012
20132014
2015
TOTAL DEVICE SHIPMENTS (M units)
Tablets & Smartbooks Smartphones
DAB Personal Media Devices (PMP)
Navigation (PND) Mobile Game Console - Gen 2
MIDs & Gadgets e-Books
Digital Still Cameras (DSC) Digital Picture Frames (DPF)
Jon
Ped
die
Re
sea
rch
The Sub or Adjacent Markets
0% 5% 10% 15% 20% 25% 30% 35%
Laptop
Smartphone
MP3 music player
Desktop
Netbook
Tablet PC
Consumer future purchase plans (PriceGrabber.com)
© Copyright Khronos Group, 2009 - Page 14
OpenGL ES Update
Tom Olson, ARM
Chair, OpenGL ES Working Group
© Copyright Khronos Group, 2009 - Page 15
What is OpenGL ES?
• OpenGL for mobile devices- Gets rid of redundant / legacy features- Extensions to make it mobile-friendly- All the functionality of the desktop API
• Versions- ES 1.0 / 1.1: fixed function- ES 2.0: shader-based via GLSL ES
• A huge success!- The dominant native 3D API for mobile devices- Widely used in other markets: STB, printers, NAV…- Now shipping > 100M units a year
© Copyright Khronos Group, 2009 - Page 16
• Available in all the latest high-end handsets
• SDKs, books are now available
• Starting point for other APIs, e.g. WebGL
OpenGL ES 2.0 is here!
…and the fun is just starting
© Copyright Khronos Group, 2009 - Page 17
Working Group Activities
• Next Generation OpenGL ES- Working Group’s main focus since mid-2009- Will be released when market needs it- Will not disrupt the growing OpenGL ES 2.0 ecosystem
• ARB / ES Convergence Group- Joint subcommittee of OpenGL and OpenGL ES working groups- Goal is to align roadmaps and avoid unnecessary divergence
• Ecosystem Support- OpenGL ES 2.0 Reference Card - KTX file format
© Copyright Khronos Group, 2010 -
Page 19
The Khronos Group Announces
http://www.khronos.org/opengles/sdk/tools/KTX/file_format_spec/
• KTX – Khronos TeXture
© Copyright Khronos Group, 2010 -
Page 20
KTX
• A new file format for storing texture images
• Oh no! Not Another image file format? Groan...!
• NO
• It is a lightweight format specifically for storing OpenGL®
textures
© Copyright Khronos Group, 2010 -
Page 21
KTX
• Designed around how textures are loaded in OpenGL.
• KTX files contain all the parameters needed for texture loading.
• The loader is very simple.
© Copyright Khronos Group, 2010 -
Page 22
Compression
• Image data can be stored in any of the compressed formats
supported by OpenGL family APIs and extensions
- e.g. ETC1, ATITC, S3TC, etc.
• Or can be stored uncompressed.
• A single .ktx file can contain any type of GL texture from a 1D
texture to an array texture with all mipmap levels.
© Copyright Khronos Group, 2010 -
Page 24
E.G. Cube Map + Mip Levels in Single KTX File
}my_cubemap.ktx
© Copyright Khronos Group, 2010 -
Page 25
Tools
• libktx
- Library of functions for writing ktx files and loading textures from
them
- Loader decompresses ETC1 compressed textures when hardware
does not support ETC1.
- Easy to use.
- E.g. app. can change from using a base level 2D texture to
providing a full mipmap pyramid simply by changing the
file it loads.
• toktx
- Create KTX files from a list of .pam, .pgm or .ppm files
- Does not compress images
• Both are open source under BSD-like license
© Copyright Khronos Group, 2010 -
Page 26
• Specification, downloads, source and documentation can all be
found on the KTX home page in new OpenGL ES SDK at:
http://www.khronos.org/opengles/sdk/tools/KTX/
© Copyright Khronos Group, 2010 -
Page 27
More Tools
• etcpack
- Creates KTX files for RGB textures compressed with ETC1.
• Available from Ericsson’s developer site:
http://devtools.ericsson.com/etc
© Copyright Khronos Group, 2010 -
Page 28
A Note About Orientation
• etcpack and toktx by default map the first pixel in the input file
to s0, t0.
• First pixel in a netpbm file is logical top-left of the image
- so image will be inverted if displayed with GL defaults
• This behavior was chosen to match existing compression tools
• etcpack and toktx provide options to change the mapping
• toktx inserts a KTXorientation key-value pair into the file that
indicates the logical orientation of the image allowing smart
applications to display the image correctly.
© Copyright Khronos Group, 2010 -
Page 29
Future
• We expect other compression tools such as The
Compressionator to add an option to output the KTX format.
• Applications will then be able to use a single file format across
all vendors proprietary texture compression schemes.
© 2004 – 2009 Graphic Remedy. All Rights Reserved
Debugging and Profiling OpenGL ES based applications using gDEBugger ES
Yaki Tebeka, CTO
Graphic Remedy
Smart phones and devices era
• Today’s smart phones and devices are
equipped with GPUs, offering high
performance graphics and low power
consumption
• Implementations are available from
PowerVR, ARM, Qualcomm, NVIDIA, …
• The OpenGL ES API is widely used over
these platforms
© 2004 – 2009 Graphic Remedy. All Rights Reserved
Debugging and profiling 3D apps
• Debugging and profiling 3D applications are
hard and time consuming tasks
• Companies find it extremely hard to deliver
robust and bug-free 3D applications
• It is almost impossible to optimize a 3D
application to fully utilize the graphic system
performance
© 2004 – 2009 Graphic Remedy. All Rights Reserved
• OpenGL, OpenGL ES and Open CL Debugger,
Profiler and Memory Analyzer
• Exposes the internal graphic system information
needed to find bugs, optimize render performance
and memory usage
© 2004 – 2009 Graphic Remedy. All Rights Reserved
Automatic errors detection
mechanism
gDEBugger can automatically break on:
• OpenGL ES errors
• gDEBugger detected errors: a unique and more
comprehensive errors detection mechanism
• Graphic memory leaks
• Redundant functions
• And more
© 2004 – 2009 Graphic Remedy. All Rights Reserved
Call stack and Source code views
• View the call stack and source code that
led to the error / OpenGL ES function call
© 2004 – 2009 Graphic Remedy. All Rights Reserved
Statistical and Redundant calls viewer
• Displays the number
of times each
OpenGL ES function
was called
• Display redundant
state changes
• Break on redundant
state change
© 2004 – 2009 Graphic Remedy. All Rights Reserved
Performance graph
Displays performance counters graphs from different
sources
• Operating Systems: Windows, Mac OS X and Linux
• GPU and driver: SGX Graphic driver, etc
• gDEBugger OpenGL ES Server
© 2004 – 2009 Graphic Remedy. All Rights Reserved
Performance Analysis Toolbar
• Turn off the graphic pipeline stages one after
the other
• If the performance improves when turning off a
certain stage, you have found a graphic
pipeline bottleneck
© 2004 – 2009 Graphic Remedy. All Rights Reserved
These commands include
• Eliminate draw commands
• Eliminate raster operations
• Eliminate fixed pipeline lights
• Eliminate texture data fetch operations
• Eliminate geometry shader operations
• Eliminate fragment shader operations
© 2004 – 2009 Graphic Remedy. All Rights Reserved
Textures and Buffers viewer
Displays all texture types, FBOs, pbuffers, Depth,
Stencil, VBOs, etc.
© 2004 – 2009 Graphic Remedy. All Rights Reserved
Shader's Source Code Editor
• Displays shader's source code
• Displays programs active uniform values
• Edit shader's source code, re-compile, link and
validate "on the fly"
© 2004 – 2009 Graphic Remedy. All Rights Reserved
Graphic Memory analysis viewer
• Displays information
about graphic memory
allocated objects
• Displays graphic
memory leaks
• Use the Object
Creation Calls Stack
to see the scenario
that led to each
object's creation
© 2004 – 2009 Graphic Remedy. All Rights Reserved
Statistical and Redundant calls viewer
• Displays a statistical
view of application
activities
• Warns about none best
practice usage
• Suggest best practice
alternatives
© 2004 – 2009 Graphic Remedy. All Rights Reserved
Questions?
www.gremedy.com
Info [at] gremedy.com
© 2004 – 2009 Graphic Remedy. All Rights Reserved
9/3/2010Confidential46
SERIOUS ABOUT SOFTWARE
OpenGL ES 2.0: 3D Graphics and In-Vehicle Infotainment (IVI)
July 29th, 2010 Petri Talala, Symbio
In Vehicle Infotainment (IVI)
Direction
• IVI systems are becoming a key differentiator between
automotive OEMs
• Huge need for graphics design and programming
expertise
• IVI systems are moving towards open platforms
– Genivi platform
– Ubuntu – Android
• IVIs heavily based on same technology as mobile
graphics
– In majority cases… same HW platforms, same OS base, same
tools
– OpenGL ES x.x, OpenVG for graphics 9/3/2010
9/3/2010
Use Cases
• Different forms of graphics and multimedia acceleration are becoming more common in automotive IVI systems– UI’s for IVI and entertainment systems
– Instrumentation• Gauges and information screens rendered with real-time graphics
• Heads Up Displays
– Video compress/decompress• Parking cameras
– Video postprocessing• Graphics overlay
• Night vision
• Navigation
– Content playback for entertainment• Video decode
• Games
9/3/2010
Use cases
Entertainment cluster:
• UI (content browsing, climate control, etc)
• Video decode for 1 or more screens from 1 or more sources
• Multiple application content
Instrumentation cluster:
• Gauges (2D/OpenVG/OpenGL ES 2.0)
• Information system UI (2D/OpenVG/OpenGL ES 2.0)
• HUD (graphics overlay)
• Night vision (graphics overlay, composition, video postprocessing)
Ecosystem Overview
9/3/2010
HW Platforms
OpenGL ES 2.0
OS /
Frameworks
Content Creation
Toolchain:
-> Model, animate
-> Export / convert-> Integrate-> Run it on target
Adaptation Work
• Customization• Optimized
booting• Optimized
usability• Scheduling
• Layering• Smooth
animations• Smooth
transitions• Automotive
Grade
Few Words about
• Non-profit industry alliance committed to IVI reference
platform based on open source IVI community
– Members such as ARM, BMW, Continental, Freescale, GM,
Intel, Marvell, NVIDIA, Renesas, Symbio, and Texas Instruments
• Genivi platform is a common SW architecture
– Linux based core services, middleware and open application
layer API
– OpenGL ES 2.0 is preferred for graphics
– Allows new compelling automotive applications and new
business models
9/3/2010
9/3/2010
Content Creation Example
HW Platform: NVIDIA Tegra2OS: LinuxOpenGL ES 2.0 based contentFull HD resolutionFPS > 30
Final Notes to Takeaway
1. IVI market is a good business opportunity with
potentially long term commitments
2. Automotive OEMs and Tier-1 suppliers urgently need
graphics expertise
3. IVIs will be mainly based on familiar HW platforms, OS,
OGLES 2.0 graphics from mobile industry
4. Usage of familiar Digital Content Creation Tools
5. Importance to select a trusted and experienced SW
Integration Partner
9/3/2010
9/3/2010Confidential
www.symbio.com
Symbio’s value is based on an extensive track record plus years of experience in demanding software integration and user interface implementation projects in mobile and embedded industry. We know how to optimize graphics rendering solutions for the best
possible visual performance and user experience, while minimizing power consumption through an optimized software stack.
Petri TalalaDirector, Business Development
[email protected]: +1 408 888 8927
60
The Khronos Ecosystem
Embedded 3D
Cross platform desktop 3D
3D Asset Interchange Format
High-level Enhanced Audio
Vector 2D
Surface and synch abstraction
Mobile OS Abstraction
3D asset interchange, 3D web content and Mobile OS portability
High-level streaming media and audio frameworks
Low-level silicon acceleration of graphics, media and parallel computation
Hundreds of man years invested by industry experts to create a coordinated visual computing ecosystem for accelerated parallel computation, 3D, video, audio and image processing on desktop,
embedded and mobile systems
Heterogeneous Parallel Programming
WebGL
3D Web Content
Streaming Media andImage Processing
High-level Steaming MediaRecording and Playback
61
OpenVG
Royalty-free open standard API
Low-level 2D vector graphics rendering API
OpenGL-style programming model
Advanced feature set enables SVG, Flash, PDF, Postscript, Java (JSR 287, 271, 226) etc.
Portable content
Map Applications
Hardware Acceleration
Applications and UI
Hardware Acceleration
SVG/Flash/Font Packages etc..
63
CPU sending data and commands to OVG HW. OpenVG rendering pipeline is in the hardware
OpenVG with Native Graphics Processor
CPU Native Vector
Graphics Hardware
64
• Many implementation alternatives
• Everything is processed on the CPU (or +FPU)
- Power consumption is often the CPU maximum power
Software Implementation
CPU
65
• 3D hardware is not tailored for tessellating arbitrary paths to triangles which causes significant pre-processing to the CPU
• Frame-rates can be discontinuous when tessellation is computed infrequently
OpenVG with a 3D GPU (tessellation approach)
Tessellation-based
implementations require complex
polygon tessellation in the software driver
CPU
GPU
66
• Path subdivision is computationally simpler than path tessellation
• Stencil buffer rasterization consumes a lot of memory bandwidth
OpenVG with a 3D GPU (the stencil approach)
Stencil-based implementations require path subdivision in the software driver
CPU
GPUTriangles are
rasterized into stencil buffer to determine path
areas
67
OpenVG with Multimedia Hardware
• Using double buffer, Fast image processing, Bilblt
Selective hardware acceleration
CPU
MM
Most of rasterization (Vector Graphics)
Implementations
Conformant Products
Takumi GV series
DMP dandelion VG,
Vivente GC series
ARM Mali series
Inst. for Information Industry DeltaVG
Imagination PowerVR SGX, MBX with VGP
Broadcom – VideoCore III
HUONE – AlexVG engine
NVIDIA – NVIDIA AP
68
Implementations
Not conformant but based on OpenVG
ShivaVG – Open Source OpenVG on OpenGL
Mazatech – AmanithVG (Software and on OpenGL ES)
Hooked Wireless – Hooked OpenVG (on OpenGL ES)
HUONE – AlexVG forge (on OpenGL ES)
69
Applications
GUI
Mobile Devices GUI – Samsung
Bitmap Graphics, Flash lite, OpenVG, Flash lite on OpenVG
Qt Rendering on OpenVG
Mapping
GPS map drawing
Flash lite player
SVG Mobile Player MMS, DCD, MPEG4-LASeR, WAP
Java Bindings - JSR226, JSR271, JSR287
70
OpenVG Roadmap
OpenVG Inari – Future Version
Investigations
Various kind of features including “shading language”
OpenVG Express – A Streamlined, refactored
Investigations
Add no new functionality
Reduce HW and SW complexity
Fix Color Space
Objectives - Supports SVGT, Flash Lite
CTS - Making it more effective and more accessible to new vendor
71
BOOK & Study Materials
AlexVG™ StudyKit – Sep. 2010 (www.alexvg.com)
Book “OpenVG Programming”
Korean Version - Jan. 2010
English Version – Jan. 2011?
Quick Ref. Card
KHRONOS Booth
72
Top Related